 MSDN Magazine April 2003

MSDN Magazine April 2003

WS-Security: New Technologies Help You Make Your Web Services More Secure

Without good security, Web Services will never reach their potential. WS-Security and its associated technologies, the focus of this article, represent the future of security for Web Services. Provided here is an overview of these emerging security standards that explains what they do, how they work, and how they get along together. Topics discussed include integrity and confidentiality and how these are provided by public key cryptography, WS-Security, and more. Some of the key components of WS-Security, such as the wsu namespace, are also covered.

David Chappell Visual Studio .NET: Building Windows Forms Controls and Components with Rich Design-Time Features

Visual Studio .NET provides support for designing rich features into your controls and components, allowing you to set properties, set form placement, inherit from base classes, and much more. So how does Visual Studio .NET do all this? What does the Windows Forms designer do? What's the difference between a control and a component? How does Visual Studio integrate your controls so that they can access features of the .NET Framework?In this article, the authors answer these common questions by building a clock control and taking the reader along for the ride. In building the control, hosts and containers are illustrated, the property browser is explained, debugging is discussed, and a general overview of the design-time infrastructure is presented.

Michael Weinhardt and Chris Sells ASP.NET: Nine Options for Managing Persistent User State in Your ASP.NET Application

ASP.NET provides many different ways to persist data between user requests. You can use the Application object, cookies, hidden fields, the Session or Cache objects, and lots of other methods. Deciding when to use each of these can sometimes be difficult. This article will introduce the aforementioned techniques and present some guidelines on when to use them. Although many of these techniques existed in classic ASP, best practices for when to use them have changed with the introduction of the .NET Framework. To persist data in ASP.NET, you'll have to adjust what you learned previously about handling state in ASP.

Steven Smith ADO.NET: Tackle Data Concurrency Exceptions Using the DataSet Object

ADO.NET provides many techniques for improving the performance of data-intensive applications and for making them easier to build. The DataSet, the hallmark of the ADO.NET object model, serves as a miniature, disconnected facsimile of a data source. While using the DataSet improves performance by reducing expensive trips to the database server, it also introduces the possibility of multiple users attempting to change the same data simultaneously, thereby generating data concurrency exceptions. This article examines the common causes behind data concurrency exceptions and presents techniques for overcoming them.

David Burgett Advanced T-SQL: Automate the Generation of Stored Procedures for Your Database

Design-time automation makes coding faster and ensures that all the procedures generated use the same naming conventions and structure. In an effort to improve their coding efficiency in a large SQL project, the authors wrote a set of design-time stored procedures that generate run-time stored procedures, and have used them in project after project ever since. Recently, the authors updated their procedures to make use of SQL Server 2000 features, including user-defined functions. This article covers the creation and execution of these dynamic T-SQL scripts to automate the coding of common database stored procedures.

Peter W. DeBetta and J. Byer Hill Data Access: Implement a Data Access Layer for Your App with ADO.NET

Implementing data access functionality is a core activity of most developers working with the .NET Framework, and the data access layers they build are an essential part of their applications. This article outlines five ideas to consider when building a data access layer with Visual Studio .NET and the .NET Framework. The tips include taking advantage of object-oriented techniques and the .NET Framework infrastructure by using base classes, making classes easily inheritable by following guidelines, and carefully examining your needs before deciding on a presentation method and external interface.

Dan Fox Editor's Note: MP3 Playlists with Visual Basic .NET

Although it's been several months since the slammer virus hit SQL Server installations, we'd like to offer a public service announcement for anyone who hasn't patched their machine yet. PATCH YOUR MACHINE NOW! We found out the hard way that it can hit any machine that runs SQL Server and is attached to the network.

New Stuff: Resources for Your Developer Toolbox

TALtech has announced a new version of its Bar Code ActiveX Control for Microsoft® Access, Excel, and Word. This new limited license version is based on TALtech's Bar Code ActiveX® PLUS version, limited for use on up to five workstations.

Theresa W. Carey Web Q&A: Consuming a DataSet in ASP.NET, XML and SQL, and More

Edited by Nancy Michell Cutting Edge: MyTracer Monitors and Traces ASP.NET Apps

The Microsoft® . NET Framework comes with a rich set of programming tools for debugging and tracing applications. I'm not talking about integrated debuggers; I'm referring to software components that you use in the development cycle.

Dino Esposito The XML Files: Web Services and DataSets

Programmers using Visual Basic® 6. 0 have long bowed to the altar of the ADO recordset. It's probably the most commonly used data structure in Windows®-based applications today. The ADO. NET DataSet is poised to play a similar role in the realm of managed Windows-based applications.

Aaron Skonnard Advanced Basics: Visual Studio .NET 2003 Usage Tips

Ken Spencer Security Briefs: Exploring S4U Kerberos Extensions in Windows Server 2003

Building Web sites that provide services external to the corporate firewall is tricky. Usually it's not desirable to grant corporate domain accounts to external clients, and from a purely practical standpoint Kerberos does not work well over the Internet due to the typical configuration of client-side firewalls.

Keith Brown Wicked Code: Supporting Database Cache Dependencies in ASP.NET

Developers love the ASP. NET application cache. One reason they love it is that ASP. NET lets them create dependencies between items placed in the cache and files in the file system. If a file targeted by a dependency changes, ASP.

Jeff Prosise C++ Q&A: Desktop Location, sscanf Equivalents in C#, and More

Paul DiLascia Resource File: ASP.NET Starter Kits

It's a well-known axiom in the world of programming: build the samples and they will come. ASP. NET is well suited for building online storefronts and communities, but a lack of easy-to-use, standardized samples has sometimes slowed adoption.


WS-Security: New Technologies Help You Make Your Web Services More Secure

WS-Security

New Technologies Help You Make Your Web Services More Secure

David Chappell
This article assumes you're familiar with XML and Security Basics
Level of Difficulty 1 2 3
SUMMARY

Without good security, Web Services will never reach their potential. WS-Security and its associated technologies, the focus of this article, represent the future of security for Web Services. Provided here is an overview of these emerging security standards that explains what they do, how they work, and how they get along together. Topics discussed include integrity and confidentiality and how these are provided by public key cryptography, WS-Security, and more. Some of the key components of WS-Security, such as the wsu namespace, are also covered.

Contents
WS-Security
Sending Security Tokens
Integrity
Confidentiality
Specifying Policies: WS-SecurityPolicy
Acquiring Security Tokens: WS-Trust
Establishing Context: WS-SecureConversation
Using SAML and Other XML-based Tokens
Conclusion

W
eb Services without effective security aren't very useful. Yet the original creators of SOAP chose to put off defining how this problem should be solved. This was a defensible decision, since getting Web Services off the ground meant keeping them simple—and providing security is seldom simple. The problem, however, can't be put off forever so Microsoft and IBM, among others, are working together to address this issue. Their efforts have resulted in a group of specs for providing Web Services security, the most important of which is WS-Security. With this article I'll provide a big-picture view of how these technologies work.

From one perspective, the task facing the creators of Web Services security looks simple. After all, effective mechanisms already exist for distributed security, including Kerberos, public key technologies, and others, so the task these creators faced wasn't inventing new security mechanisms. Instead, their goal was to define ways to use what already existed in a Web Services world, a world built on XML and SOAP. There are several security-related specifications, which include the following:

  • WS-Security, the foundation for all of the other specs. It defines extensions to SOAP that allow passing security tokens which securely identify and authenticate clients, ensuring message integrity, and providing message confidentiality. An accompanying spec, the Web Services Security Addendum, adds some details and fixes to the original document.
  • WS-SecurityPolicy, which specifies how to define security assertions that clearly state a Web Service's preferences and requirements.
  • WS-Trust, which defines how to acquire security tokens.
  • WS-SecureConversation, which defines how to create a context for a particular conversation with a Web Service and how to create keys that can be used in that context.
  • Web Services Security Profile for XML-based Tokens, which defines how XML-based technologies such as the Security Assertion Markup Language (SAML) and the eXtensible rights Markup Language (XrML) can be used with WS-Security. It also wins the prize for longest spec name.

Taken as a group, these specs lay the foundation for a usable, interoperable, and quite complete approach to providing security for Web Services.

WS-Security

The fundamental technology for adding security to SOAP is defined by WS-Security. Its ambitious goal is to provide end-to-end message-level security for SOAP messages, and yet the WS-Security spec isn't especially hefty, weighing in at just over 20 pages. This is because WS-Security defines very little new technology but instead defines a way to use existing security technology with SOAP.

Perhaps the most fundamental thing WS-Security defines is a <Security> element that lives inside the SOAP header. The structure of a SOAP message that includes this element is shown in Figure 1. As this simple skeleton shows, WS-Security defines its own namespace. Still, the creators of WS-Security defer to existing standards and technologies wherever possible, focusing instead on specifying how to use them with SOAP.

Figure 1 Security Element (Close Figure 1)
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
 <s:Header>
  <wsse:Security>
     •••
  </wsse:Security>
 </s:Header>
 <s:Body>
  •••
 </s:Body>
</s:Envelope>

WS-Security describes how to attain integrity, which allows a receiver to be sure that data it receives in a message was not modified in transit, and confidentiality, which ensures that data can't be read in transit. It also describes how to send security tokens, such as a username/password combination, a Kerberos ticket, or an X.509 certificate. The next three sections describe how WS-Security addresses each of these areas.

Sending Security Tokens

The most basic question the receiver of a message can have is: who am I talking to? The question is simple, but the answer is not. There are many different ways of indicating identity on a network (a username, Kerberos ticket, X.509 certificate, and so on) and there are also different ways of authenticating that identity. With a username, for example, you might provide an accompanying password to prove that this really is your identity. A Kerberos ticket, by contrast, is encrypted by its issuer using a key that the ticket's receiver can verify is correct. Additionally, digital signature might be sent along with a certificate to authenticate your identity. (For more information on digital signatures, take a look at the sidebar "What's a Digital Signature?")

Wisely, the creators of WS-Security chose to rely on more than one approach for conveying and authenticating identity. In fact, WS-Security doesn't define how to do authentication at all. Instead, the spec defines how a variety of different security tokens can be conveyed within a SOAP header's Security element. The receiver of a token can use the information it contains in any way it likes. The most common use is likely to be authenticating the identity of the token's sender, but tokens can also be used in other ways.

While it allows any type of security token, the WS-Security specification explicitly defines three options. The simplest (although not always the most secure) option is to send a security token containing a username and password. To allow this, WS-Security defines a <UsernameToken> element that can contain a username and password. Here's a simple example of a Security element containing just a <UsernameToken>:

<wsse:Security
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
 <wsse:UsernameToken>
  <wsse:Username>Diana</wsse:Username>
  <wsse:Password>mY5ecRet</wsse:Password>
 </wsse:UsernameToken>
</wsse:Security>

More powerful options are also defined, such as sending a digest version of the password. Because sending unencrypted passwords across a network isn't a very effective authentication mechanism, the UsernameToken element is most likely to be used to authenticate SOAP messages sent across an encrypted connection, such as one that uses Secure Sockets Layer (SSL).

A second option defined by WS-Security is to send a security token containing a Kerberos ticket. Like all security tokens, this ticket is sent in the SOAP header as part of the Security element. Here's an example:

<wsse:Security
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
 <wsse:BinarySecurityToken
  ValueType="wsse:Kerberosv5ST"
  EncodingType="wsse:Base64Binary">
   QMwcAG ...
 </wsse:BinarySecurityToken>
</wsse:Security>
As this snippet shows, a Kerberos ticket is sent using the more general <BinarySecurityToken> element. This element's ValueType attribute indicates that this is a Kerberos Version 5 service ticket, which is used to authenticate a client to a particular service. (When a ticket-granting ticket is sent, the ValueType is wsse:Kerberosv5TGT.) The ticket itself, only the first few bytes of which are shown, is represented using base64 encoding, although an option for hexadecimal encoding is also defined.

The third option WS-Security defines for representing security tokens, X.509 certificates, also relies on the <BinarySecurityToken> element. Here's an example:

<wsse:Security
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
 <wsse:BinarySecurityToken
  ValueType="wsse:X509v3"
  EncodingType="wsse:Base64Binary">
   KkFPle ...
 </wsse:BinarySecurityToken>
</wsse:Security>

As you can see, not much is different from the previous Kerberos ticket example. This time, the ValueType attribute of the <BinarySecurityToken> indicates that what's being sent is an X.509 certificate. Base64 encoding is used once again, and the certificate itself makes up the content of this element, just as the Kerberos ticket did in the earlier case.

Along with these three options for embedding a security token directly in the header of a SOAP message, WS-Security also defines a <SecurityTokenReference> element. As its name suggests, this element contains a URI that indicates where a security token can be found. This allows the message's receiver to pull the token from wherever it resides, such as a named location on the Internet. This element can also be used to refer to a security token contained within this message's SOAP header, a feature whose usefulness will become clear in the next section.

WS-Security takes an admirably flexible approach to conveying and authenticating identities, yet there's one important implication here: two systems can both conform to WS-Security, but still be incapable of authenticating one another. One system might support only Kerberos, for instance, while the other allows only digital signature-based authentication using X.509 certificates. Just agreeing to use WS-Security isn't enough. Some agreement must also be in place about exactly what kinds of security tokens will be used.

Integrity

Conveying and authenticating identity with SOAP messages is a multifaceted problem, one with many possible solutions. Accordingly, WS-Security takes a commendably broad approach. Providing integrity for SOAP messages, however, is much cleaner. The solution is to use some kind of digital signature, and fortunately, a W3C standard already exists that addresses this area. Called XML Signature, it provides a detailed mechanism for digitally signing XML documents. Rather than reinvent what already exists, WS-Security simply points to this existing specification and fills in a few details on how to use it with SOAP messages.

The main thing defined by the XML Signature specification is a <Signature> element whose contents include both the digital signature itself and information about how this signature was produced. Although there are a few options, a typical instance of <Signature> as used with SOAP contains the <SignedInfo> and <KeyInfo> elements.

The <SignedInfo> element describes the information that is being signed. This element itself contains several subelements, each of which specifies something about the signed information. The most important of these include the following elements.

  • <CanonicalizationMethod> identifies the algorithm that was used to convert this XML document into a standard form before the signature was generated. This process is necessary because for digital signatures to work, each party's view of the signed document must be identical bit-for-bit. Yet without canonicalization, two XML documents that are semantically equivalent can be different at the bit level. One may represent line breaks with a carriage return/line feed pair, for instance, while another may use just a line feed. Canonicalization transforms an XML document into a standard form, ensuring that both parties view a document in the exact same way before signing it.
  • <SignatureMethod> identifies the algorithm used to create the digital signature. Both the XML Signature standard and WS-Security require support for the Digital Signature Standard, which uses Secure Hash Algorithm (SHA)-1 to create a message digest together with the DSA algorithm, and also recommend support for using SHA-1 together with the RSA algorithm.
  • <Reference> identifies the information being signed, along with any transformations that were applied to it. When used with SOAP, this part of the signature typically references the signed parts of this SOAP message, such as the body and parts of the header. The <Reference> element also contains the base64-encoded message digest that was generated from the signed information and specifies the algorithm used to create that digest, such as SHA-1.
  • <SignatureValue> contains the bytes that make up the digital signature.
  • <KeyInfo> indicates what key should be used to validate this signature. The key can be identified in a variety of ways, such as referencing a certificate that contains the appropriate public key. This element can even be omitted entirely, since the recipient may know what key to use by some other means.

Given this basic understanding of XML Signature's <Signature> element, it's possible to see how WS-Security uses it. Figure 2 is an example of how a SOAP message containing a digital signature might look.

Figure 2 SOAP Message with Digital Signature (Close Figure 2)
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext"
 xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility">
 <s:Header>
  <wsse:Security>
   <wsse:BinarySecurityToken
    ValueType="wsse:X509v3"
     EncodingType="wsse:Base64Binary"
     wsu:Id="X509Cert">
     KkFPle ...
   </wsse:BinarySecurityToken>
   <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
    <ds:SignedInfo>
     <ds:CanonicalizationMethod
       Algorithm="http://www.w3.org/2001/10/xml-exc-c14N"/>
     <ds:SignatureMethod
       Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
     <ds:Reference URI="#MessageBody">
      <ds:DigestMethod
        Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
      <ds:DigestValue>
        aOb4Luuk...
      </ds:DigestValue>
     </ds:Reference>
    </ds:SignedInfo>
    <ds:SignatureValue>
      A9qqIrtE3xZ...
    </ds:SignatureValue>
    <ds:KeyInfo>
     <wsse:SecurityTokenReference>
      <wsse:Reference URI="#X509Cert"/>
     </wsse:SecurityTokenReference>
    </ds:KeyInfo>
   </ds:Signature>
  </wsse:Security>
 </s:Header>
 <s:Body wsu:Id="MessageBody">
  •••
 </s:Body>
</s:Envelope>

In this example, the Header's <Security> element begins with a <BinarySecurityToken>. This element is nearly identical to the X.509 certificate example shown in the previous section. The only difference is that this element uses the Id attribute defined by the Web Services Security Addendum to provide a name for this <BinarySecurityToken>. (Take a look at the sidebar "The wsu Namespace" for more information.) The Id attribute's value, which here is "X509Cert", allows it to be referenced from elsewhere in this header. As you'll see shortly, this is quite useful when a digital signature is also contained in this message.

The <Signature> element defined by the XML Signature standard appears next, containing the subelements described earlier: <SignedInfo>, <SignatureValue>, and <KeyInfo>. Within the <SignedInfo> element appear the identifiers for the algorithms used to canonicalize this XML document and to create its digital signature. The <Reference> element has a URI attribute that specifies exactly what is being signed. In this example, just the body is signed, and so this URI matches the value of the Id attribute associated with the message's <Body> element. The Reference element also contains the two subelements: <DigestMethod> and <DigestValue>. The first of these identifies the algorithm used to create the message digest for this digital signature, while the second contains the message digest value itself. Because only the body is signed in this example, this digest was computed from the canonicalized version of the SOAP message's <Body> element.

The next <Signature> subelement, <SignatureValue>, contains the bytes of the digital signature itself, created by encrypting the message digest with the signer's private key. Finally, the last subelement in <Signature>, <KeyInfo>, identifies where to find the public key that should be used to validate this signature. In this example, that key appeared earlier in the message (in the security token), and it's referenced here through <SecurityTokenReference>. The key is identified using its Id attribute, "X509Cert".

Ensuring data integrity is critically important for many Web Services applications, and having a well-defined and widely accepted approach to providing this service with SOAP is very useful. Digital signatures created using public key technology are also useful for authentication because providing a valid signature along with a trusted certificate proves knowledge of a particular private key and so authenticates one's identity. Yet creating digital signatures for XML documents isn't the simplest thing in the world. Exactly what is being signed must first be identified, then put in canonical form, then actually signed. It can be even more complex than the example shown here, since WS-Security allows the creation of multiple signatures for the same message, each possibly created by a different recipient in the message's path, and each perhaps covering different or overlapping parts of the message. Building all of this according to the W3C XML Signature standard makes good sense, yet SOAP headers that use this aspect of WS-Security are bound to be bigger and more complex than those that don't need this particular security service.

Confidentiality

Keeping data safe from prying eyes while it's in transit is important for a significant number of applications. Accordingly, the third service defined by WS-Security is confidentiality, providing a way to encrypt some or all of a SOAP message before it's transmitted. In the same way they addressed the question of integrity, the authors of WS-Security chose to build on existing standards for confidentiality rather than create something new. Once again, the existing standard they chose, a spec called XML Encryption, came from the W3C. Using this standard, WS-Security allows encrypting all or part of a SOAP message's header information, its body, and any attachments. The XML Encryption spec is not a terribly complex document. WS-Security, for example, uses only three of the XML elements defined by this standard: <EncryptedData>, <EncryptedKey>, and <ReferenceList>.

The most important of these is certainly <EncryptedData>. As its name suggests, this element contains the actual encrypted data in a subelement called <CipherData>. This element can also contain subelements which indicate the encryption algorithm that was used, the key that was used in order to perform the encryption, and more. Figure 3 shows an example of how a simple SOAP message might look if its sender chose to encrypt the message body as defined by WS-Security.

Figure 3 SOAP Message with Encrypted Body (Close Figure 3)
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
 xmlns:xenc="http://www.w3.org/2001/04/xmlenc#">
 <s:Body>
  <xenc:EncryptedData>
   <EncryptionMethod
     Algorithm='http://www.w3.org/2001/04/xmlenc#tripledes-cbc'/>
   <ds:KeyInfo>
    <ds:KeyName>
      CN=Key13, C=US
    </ds:KeyName>
   </ds:KeyInfo>
   <xenc:CipherData>
    <xenc:CipherValue>
      r5KipsDV . . .
    </xenc:CipherValue>
   </xenc:CipherData>
  </xenc:EncryptedData>
 </s:Body>
</s:Envelope>

Note first that no SOAP header appears in the message. While this isn't likely in a real application, it is possible to use XML Encryption as defined by WS-Security without the Security element or any other Header elements. In this simple example, the message's body contains only an <EncryptedData> element with three subelements. The first of these, <EncryptionMethod>, indicates what algorithm was used to encrypt the data. In this case, the symmetric algorithm TripleDES was chosen, and so the same key is used to encrypt and decrypt the data. The next subelement is <KeyInfo>, borrowed from XML Signature. Here, it is assumed that both sides already know all of the keys available for use in encryption and decryption, so the only thing required is to inform the receiver of which one was chosen. The key's name could be anything, but one possibility is that keys be named with LDAP-style distinguished names. Accordingly, this example specifies the key's name as "CN=Key13, C=US". Whatever name is used, the important thing is that both sides understand exactly which key this name references. Finally, the <CipherData> subelement appears, carrying within its <CipherValue> subelement the base64 encoding of the actual encrypted data.

It's also possible to convey an encrypted key in the same message that carries data that was encrypted using that key. A common example of this is when data is encrypted using a symmetric key, as in the TripleDES example in Figure 3, then that symmetric key is encrypted using the public key of the message's recipient and sent along with the data. When the message arrives, the recipient can use its private key to decrypt the embedded symmetric key, then use this symmetric key to decrypt the actual data. To send an encrypted key, XML Encryption defines an element with the obvious name of <EncryptedKey>. For use with SOAP messages, WS-Security specifies that this element should appear in the <Security> header. Accordingly, a SOAP message that carries both encrypted data and the encrypted symmetric key needed to read that data might look like Figure 4.

Figure 4 Encrypted SOAP Message with Key (Close Figure 4)
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext"
 xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility"
 xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
 xmlns:xenc="http://www.w3.org/2001/04/xmlenc#">
<s:Header>
 <wsse:Security>
  <xenc:EncryptedKey>
   <xenc:EncryptionMethod
    Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5"/>
   <ds:KeyInfo>
  <ds:KeyName>
   CN=Key13, C=US
  </ds:KeyName>
   </ds:KeyInfo>
   <xenc:CipherData>
  <xenc:CipherValue>
   ir4DfG . . .
  </xenc:CipherValue>
   </xenc:CipherData>
   <xenc:ReferenceList>
    <xenc:DataReference URI="#EncryptedBody"/>
   </xenc:ReferenceList>
  </xenc:EncryptedKey>
 </wsse:Security>
</s:Header>
<s:Body>
 <xenc:EncryptedData wsu:Id="EncryptedBody">
  <xenc:EncryptionMethod
   Algorithm='http://www.w3.org/2001/04/xmlenc#tripledes-cbc'/>
  <xenc:CipherData>
   <xenc:CipherValue>
r5KipsDV . . .
   </xenc:CipherValue>
   </xenc:CipherData>
  </xenc:EncryptedData>
 </s:Body>
</s:Envelope>

Unlike the unrealistically simple example in Figure 3, the SOAP message in Figure 4 contains a header with a Security element. This element contains just the subelement <EncryptedKey>, defined in the XML Encryption standard. The format of <EncryptedKey> is much like <EncryptedData> and in this example, it has the three familiar subelements: <EncryptionMethod>, <KeyInfo>, and <CipherData>. What's being encrypted here is a symmetric key, so <EncryptionMethod> describes how this key was encrypted rather than how the actual data was encrypted. Since we're assuming that this is a symmetric key encrypted using the recipient's public key, this element indicates that the RSA algorithm was used to encrypt this key. The <KeyInfo> element names the symmetric key that's conveyed as part of this <EncryptedKey>, and since we're assuming it's the same key as in the previous example, it has the same name. Unlike the earlier <EncryptedData> element, the <CipherData> element in this <EncryptedKey> contains an encrypted symmetric key rather than any user data.

The <EncryptedKey> element has a new subelement as well, one that wasn't present in <EncryptedData>. This element, <ReferenceList>, plays the important role of associating this key with the data that was encrypted using it. It contains a URI that identifies the portion of this SOAP message that was encrypted using the key contained in this <EncryptedKey> element. In Figure 4, the entire SOAP body is encrypted with this key and so the rather unimaginative name, "EncryptedBody," is used.What's a Digital Signature?

M ost of the time, the term "digital signature" refers to something created using public key technology. This technology assumes that you are given a pair of keys: a private key, which only you know, and a corresponding public key, which you can freely reveal to anyone. The cool thing about these two keys is that anything encrypted using one key can be decrypted only with the other one.

If I want to attach a digital signature to something, I first compute a special kind of checksum called a message digest on the information I want to sign. I then encrypt this digest, which is typically around 20 bytes, using my private key. The result is a digital signature for this information. If I send you the information along with this signature, you need know only my public key to validate the signature. To do this, you compute a message digest on the received data, and then use my public key to decrypt the digital signature. If the results of these two operations are identical, you can be certain both that the data wasn't changed in transit (integrity) and that the data was signed by me (authentication). Of course, you have to be certain that this really was my public key, so you'll probably extract it from a certificate whose issuer you trust.

This is the classic definition of a digital signature. Interestingly, though, that's not exactly what is meant in XML Signature. While XML Signature certainly allows using public key-based digital signatures, it also supports the idea of encrypting a message digest with a symmetric (secret) key. This approach still provides integrity (it's what Kerberos uses, for example), but it doesn't provide the same level of authentication as a public key-based signature. Since two or more parties know the symmetric key, data encrypted using it could have come from any one of them. Accordingly, this kind of signature doesn't allow you to learn exactly who created it. Still, it's a useful option, so WS-Security supports this non-public key-based approach to providing data integrity as well as more conventional digital signatures.

The SOAP body itself looks much as it did in Figure 3. One difference is that the beginning tag of the <EncryptedData> element now has an Id attribute with the name "EncryptedBody." As just described, this is how the data this element carries is associated with the key used to encrypt that data. The other difference is that since the key used to encrypt this element's contents is clearly identified, there's no need for the <KeyInfo> element in <EncryptedData>.

As defined by WS-Security, XML Encryption can be used in a number of ways with SOAP messages. You can encrypt the entire body, the body and certain header elements, just parts of the body, or even attachments sent with the message. Different receivers of a SOAP message can add their own encrypted headers or decrypt and process parts intended solely for them. Because the requirements for confidentiality in Web Services are so diverse, the standard that defines how to do it must also be quite versatile.

WS-Security does a good job of specifying the basics for providing security with SOAP, but it doesn't address everything. As a result, several other specifications have been created that fill in some of the gaps. The following sections take a look at these documents and the technology they define.

Specifying Policies: WS-SecurityPolicy

Whenever security mechanisms are being used, some kind of security policies must also exist. These policies clarify the specific security requirements for a particular situation. With a Web Service using WS-Security, for example, many details need to be made clear. Does this service accept Kerberos tickets for authentication, X.509 certificates and digital signatures, or something else? What digital signature algorithms are accepted? What encryption algorithms are supported for providing confidentiality? Answering these kinds of questions means defining security policies.

Policies are useful in areas other than security, of course, and so Microsoft, IBM, and others have created the WS-Policy specification. This document defines a general approach to specifying policies of all kinds for Web Services. WS-SecurityPolicy builds on this foundation, defining XML elements that can be used to specify security-related policies. These elements are referred to as assertions because they allow a Web Service to specify its policies in an unambiguous way. The assertions defined by WS-SecurityPolicy include the following.

  • <SecurityToken> allows a Web Service to specify what types of security tokens it will accept and, optionally, which issuers of those tokens it trusts.
  • <Integrity> allows a Web Service to specify various options about the digital signatures it will accept, such as the algorithm used to create a message digest.
  • <Confidentiality> allows a Web Service to specify options about how encryption should be performed, such as the encryption algorithm used.
  • <Visibility> allows a Web Service to specify that certain portions of a message must be unencrypted.

Exactly how security policies are conveyed from their creator to any interested clients is not defined by WS-SecurityPolicy. The XML elements that express policies might be sent in a SOAP message or conveyed in some other way. WS-Policy does define a general <Policy> element, however, that can contain one or more policy assertions. Here's a simple example of how a <SecurityToken> assertion might look for a Web Service that requires a Kerberos ticket for authentication:

<wsp:Policy
 xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext">
 <wsse:SecurityToken wsp:Usage="wsp:Required">
  <TokenType> wsse:Kerberosv5ST</TokenType>
 </wsse:SecurityToken>
</wsp:Policy>

This <Policy> element specifies namespaces defined in both WS-Policy and WS-Security, and contains just the <SecurityToken> subelement. This subelement's usage is marked as Required, which means that the Web Service requires adherence to this policy. In other words, as the <TokenType> element indicates, this particular Web Service will now accept only Kerberos tickets as the basis for authentication.

A single <Policy> element can contain policies about security tokens, integrity, and other things. There are also several more options, including ways to specify relative preferences. For example, a Web Service can define a <SecurityToken> that allows clients to authenticate using either Kerberos or X.509 certificates, then indicate that it prefers Kerberos. Whatever options are used, however, the value of policies is apparent. By providing an unambiguous way for a Web Service to state its security requirements, they make clear what clients must do to access that service.

Acquiring Security Tokens: WS-Trust

Security tokens are fundamental to WS-Security, yet they've just magically appeared and no mechanism has been defined to give them out. The truth is that just as various kinds of security tokens already exist, software services already exist that are capable of handing them out. To get a Kerberos ticket, for instance, you contact a Kerberos Key Distribution Center (KDC), while to get an X.509 certificate, you might contact a certification authority (CA). While WS-Security defines how to wrap Kerberos tickets and X.509 certificates inside standard XML for transmission in a SOAP message, it doesn't define a way to use SOAP to talk to a KDC, a CA, or any other service capable of handing out a security token.

Figure 5 WS-Trust
Figure 5 WS-Trust

WS-Trust fixes this problem. The main thing this spec defines is a mechanism for requesting security tokens from some security token service, such as a KDC or a CA. Figure 5 illustrates this idea. Suppose a client wants to invoke both Web Service A and Web Service B. Suppose further that Web Service A requires a Kerberos ticket for authentication, while Web Service B demands a digital signature and matching X.509 certificate. As the figure shows, the client might first make a SOAP request to Security Token Service X, which is in fact a KDC, requesting a Kerberos ticket wrapped inside a <BinarySecurityToken>. It can then invoke Web Service A, passing this newly acquired token in the <Security> element of the SOAP message. Next, the client can make a SOAP request to Security Token Service Y, which is a CA, requesting an X.509 certificate. Once this is returned, the client can invoke Web Service B, passing this certificate and a matching digital signature.

WS-Trust defines what requests to a security token service look like and how responses are sent back. The interaction relies on two elements: <RequestSecurityToken> and <RequestSecurityTokenResponse>. The basic idea is to send a SOAP message containing a <RequestSecurityToken> as its body, then get back a SOAP message containing a <RequestSecurityTokenResponse>, complete with the new security token. Figure 6 shows a simple example of a request message.

Figure 6 Request Message (Close Figure 6)
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext"
 xmlns:wsu="http://schemas.xmlsoap.org/ws/2002/07/utility">
 <s:Header>
  <wsse:Security>
   <wsse:UsernameToken wsu:Id="MyCert">
    <wsse:Username>Diana</wsse:Username>
    <wsse:Password>mY5ecRet</wsse:Password>
   </wsse:UsernameToken>
  </wsse:Security>
 </s:Header>
 <s:Body>
  <wsse:RequestSecurityToken>
   <wsse:TokenType>wsse:X509v3</wsse:TokenType>
   <wsse:RequestType>wsse:ReqIssue</wsse:RequestType>
   <wsse:Base>
    <wsse:Reference URI="#MyCert"/>
   </wsse:Base>
  </wsse:RequestSecurityToken>
 </s:Body>
</s:Envelope>

The Security header contains a <UsernameToken> to authenticate this request. While the simple choice used here might be reasonable if this request was sent over an encrypted SSL connection, sending this request in the clear would require a more robust security token, probably one that included both a digital signature and a timestamp. The message body contains the <RequestSecurityToken> element, which requests that an X.509 certificate be issued. Assuming the request is successful, the body of the response returned by the security token service might look like this:

<s:Body>
 <wsse:RequestSecurityTokenResponse>
<wsse:RequestedSecurityToken>
   <wsse:BinarySecurityToken
     ValueType="wsse:X509v3"
     EncodingType="wsse:Base64Binary">
     KkFPle ...
   </wsse:BinarySecurityToken>
  </wsse:RequestedSecurityToken>
 </wsse:RequestSecurityTokenResponse>
</s:Body>

The requested certificate is returned, wrapped in the <BinarySecurityToken> element defined by WS-Security. Although it's not shown in this example, it's also possible that the private key associated with this certificate's public key would be returned as well. To allow this, WS-Trust defines a <RequestedProofToken> element to contain this or analogous information for other types of security tokens.

WS-Trust also defines quite a few extensions to the basic <RequestSecurityToken> element. Among them are options for requesting that specific algorithms and key sizes be used in constructing the requested security token. For instance, the requester can ask that RSA with 1024-bit keys should be used for creating digital signatures. There is also a mechanism for requesting a security token on behalf of someone other than the requester. And a third extension offers a way to specify the lifetime of the security token and whether that lifetime can be renewed.

WS-Trust also defines a challenge/response mechanism that can optionally be used when a security token is requested from a security token service. While a straightforward authentication mechanism such as the one shown in Figure 5 might be fine in some cases, it's possible that a security token service might want more proof of who it's talking to. What if, as in the example just shown, the request contains no timestamp? How can the security token service be certain that this isn't a replay of an earlier request that is now being sent by an attacker? In order to prevent this, the security token service might challenge the requester to digitally sign a specific set of information, then validate the signature before issuing the requested security token.

Establishing Context: WS-SecureConversation

Sometimes all that's required to solve a problem is a simple exchange of one or two SOAP messages. Invoking a single remote operation, for example, might require nothing more than this. But suppose an application needs to exchange lots of SOAP messages with some other application, perhaps by invoking a series of operations. In cases like this, it can be useful to create some kind of shared security context between the two applications. One common use of this shared context is to define a lifetime for an encryption key that will be used for the context's duration. For example, two communicating parties might want to create a symmetric key, then use it to encrypt the information they exchange over the lifetime of a particular security context. (A non-Web Services example of this is SSL, which creates a new key to encrypt the bytes transferred within each session.)

WS-SecureConversation defines XML types and interactions that allow the establishment of a security context and the creation of keys that are specific to that context—that is, keys that are used only for a particular session. The foundation for these mechanisms is the <SecurityContextToken> element. As defined by this spec, every security context has a name and a shared secret, and so a <SecurityContextToken> can contain an <Identifier> subelement that uniquely names this context together with one or more encrypted keys for use in this context.

WS-SecureConversation defines three possible approaches to creating a <SecurityContextToken>. First, a <SecurityContextToken> can be requested from some security token service. To do this, the requester sends a SOAP message containing a <RequestSecurityToken> element to a security token service in the body, just as if an ordinary security token were being requested. The service responds with a SOAP message, the body of which contains a <RequestSecurityTokenResponse> element that contains the requested <SecurityContextToken> and a <RequestedProofToken> containing a shared secret for this new context, such as an encrypted symmetric key.

Second, a <SecurityContextToken> can be created by one of the communicating parties and sent to the other. In this case, the creator of <SecurityContextToken> sends the new token in a SOAP message containing a <RequestSecurityTokenResponse>. (There is no <RequestSecurityToken> sent first). This is the same message received from the security token service in the previous case, so it contains both the new <SecurityContextToken> and the <RequestedProofToken>.

Third, a <SecurityContextToken> can be created through negotiation between the parties that will use the context. One party sends the other a SOAP message containing a <RequestSecurityToken>, then receives a response that contains a <RequestSecurityTokenResponse>. Although it's not required, this exchange is likely to rely on the WS-Trust-defined challenge/response negotiation process described earlier. If the negotiation succeeds, the result is that both parties wind up with a new <SecurityTokenReference> and <RequestedProofToken>.

However it's created, a security context always contains a shared secret, such as a symmetric key. This key can be used directly for encrypting or signing messages, but WS-SecureConversation recommends against this. Instead it's more secure to derive one or more keys from the security context's shared secret and then use a derived key for the communication. The spec defines one algorithm for creating derived keys, based on the approach used in the Transport Level Security (TLS) spec. Other approaches can also be used to generate derived keys, as the spec doesn't require the use of the algorithm that it defines.

Once a security context has been created, all that's required is to identify the context in each SOAP message. Each side of the communication already knows which key (or keys) to use in processing this message, since the context implies this. Once the context exists, then the Security element of a SOAP message using it will contain a <SecurityContextToken> element with this context's unique identifier. By examining this identifier, the message's receiver can determine what security context is being used.

Security contexts are nice to have, as shown by their use today in SSL and Kerberos, among other security technologies; WS-SecureConversation makes them available to Web Services applications.

Using SAML and Other XML-based Tokens

As I described earlier in this article, WS-Security defines two XML elements for representing security tokens. But what about other kinds of security tokens? In particular, what about security tokens that are already defined using XML? WS-Security itself doesn't address how this kind of token should be represented, so a specification called Web Services Security Profile for XML-based Tokens remedies this problem. It defines a general approach to using XML-based security tokens with WS-Security, and then specifies how to use two types of existing XML tokens.

The first type of XML security token addressed by this spec is defined using the Security Assertion Markup Language (SAML). SAML is likely to be a reasonably popular technology, but understanding it is nontrivial. (If you're an insomniac, then a copy of the SAML spec works like a charm when sleep won't come.) As its name suggests, the language allows you to express a wide range of assertions about subjects, such as people or applications. These assertions can be about authentication (how it was done), or authorization (whether access was granted to a particular resource), or something else. The assertions themselves can be authenticated using XML Signature, although this isn't required.

Rather than define another type of security token element, SAML assertions can be inserted directly into a SOAP header's <Security> element. Unlike Kerberos tickets or X.509 certificates, SAML assertions are already expressed in XML, so this is easy to do. The slightly simplified result is shown in Figure 7. As this example illustrates, a SAML Assertion element can be dropped in its native form into the Security element. Although it's not shown in this example, the AssertionID attribute can be used to associate this SAML security token with a digital signature or something else within this SOAP message.

Figure 7 SAML Assertion (Close Figure 7)
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope
 xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/12/secext"
 xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion">
 <s:Header>
  <wsse:Security>
   <saml:Assertion
    MajorVersion="1"
    MinorVersion="0"
    AssertionID="SecurityToken-ef375268"
    Issuer="dcatignani"
    IssueInstant="2003-05-23T11:04:15.8357639-08:00">
    •••
   </saml:Assertion>
  </wsse:Security>
 </s:Header>
 <s:Body>
  •••
 </s:Body>
</s:Envelope>

The other type of XML security token addressed by the Web Services Security Profile for XML-based Tokens spec is XrML. XrML allows you to express rights about digital content or services, such as the right to access a particular Web Service or to play a specific audio file. It defines a <license> element that can be dropped into a SOAP header's Security element in the same manner as a SAML <Assertion>.

SAML and XrML are relatively new technologies, so you might not see these options used with SOAP for a while yet. Still, either or both of them may become very popular over time, and other XML-based tokens are also likely to emerge. Providing a standard approach for including those tokens in a SOAP message makes sense.The wsu Namespace

T he Web Services Security Addendum defines quite a few extensions (and some fixes) to the original WS-Security spec. The new XML elements this document defines are contained in a Web Services utility namespace, identified by http://schemas.xmlsoap.org/ws/2002/07/utility and conventionally given the prefix wsu. This namespace includes two things that are most important for WS-Security. The first is an Id attribute that can be applied to various elements. In WS-Security, it plays an important role in both signatures and encryption. The second is a <Timestamp> element that can be sent within a SOAP message's header. Among other things, this element can contain subelements that indicate when a message was created, when any intermediaries received it, and when it should expire. The receiver of a message might use this information to decide whether a message is stale and should be ignored, to detect replays of the same message, or for some other purpose. No particular mechanism is mandated for synchronizing clocks, however, so the systems involved must deal with this problem in some other way.

Getting complicated specs right the first time is essentially impossible, so it's not too surprising that some things were added after the first version. Chances are that you'll see more additions as the understanding of the problems to be solved improves.

Conclusion

Effective security for Web Services is essential. Given that most of what's needed is already in place (and the problem is simply a matter of mapping this existing security technology to XML and SOAP), you might think that WS-Security and its associated specs would be very simple (and this article would be very short). Yet accommodating the diverse security mechanisms in use today, along with allowing for those that will appear tomorrow, requires a nontrivial set of technology. Once this technology is in place, the world of secure and interoperable Web Services that we'd all like to see can become a reality. I don't know about you, but I can't wait for this day to arrive.

For related articles see:
http://msdn.microsoft.com/security

For background information see:
WS-Security Specification Index Page
ASP.NET Security: An Introductory Guide to Building and Deploying More Secure Sites with ASP.NET and IIS
MSDN Magazine Security issue

David Chappellis Principal of Chappell & Associates in San Francisco, CA. He has given keynotes and seminars on Microsoft technologies in more than 30 countries, and his most recent book, Understanding .NET, was published last year by Addison-Wesley.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Visual Studio .NET: Building Windows Forms Controls and Components with Rich Design-Time Features

Visual Studio .NET

Building Windows Forms Controls and Components with Rich Design-Time Features

Michael Weinhardt and Chris Sells
This article assumes you're familiar with Visual Studio .NET and C#
Level of Difficulty 1 2 3
Code download available at: Design-TimeControls.exe (286KB)
SUMMARY

Visual Studio .NET provides support for designing rich features into your controls and components, allowing you to set properties, set form placement, inherit from base classes, and much more. So how does Visual Studio .NET do all this? What does the Windows Forms designer do? What's the difference between a control and a component? How does Visual Studio integrate your controls so that they can access features of the .NET Framework?

In this article, the authors answer these common questions by building a clock control and taking the reader along for the ride. In building the control, hosts and containers are illustrated, the property browser is explained, debugging is discussed, and a general overview of the design-time infrastructure is presented.

Contents
Design-time Architecture
Hosts and Containers
Sites
Testing a Component
Customizing the Toolbox
Debugging
Components versus Controls
Design Mode
Attributes
Property Browser
Code Serialization
Batch Initialization
Extender Controls
Where Are We?

W
hen building a Windows® Forms application, you create forms, add components and controls, consume events, and write custom code. All of this is done with the intention of responding to user demands at run time. Likewise, at design time, components and controls themselves have to respond to developer demands. Towards that end, the Microsoft® .NET Framework includes rich design-time support, offering a bevy of interfaces, base classes, and attributes to increase the flexibility and usability of your controls and components, from simple form placement and property settings to full UI editing support.

This article offers an introduction into the world of design-time functionality in the .NET Framework. We'll cover the basics of the design-time architecture and some of the simpler things you can do to influence how your components and controls interact with it. We'll walk through building and testing a clock control to demonstrate the various design-time features and services. In a future companion article, we'll delve into some more complicated features. The goal of this series is to provide an overview of the .NET Framework's extensive design-time support and infrastructure. For in-depth coverage of specific features, take a look at Shawn Burke's articles on MSDN® Online.

Design-time Architecture

Design mode is activated the moment a form is opened for editing. It can involve activities like resizing, setting the border style, changing the caption, and adding controls and components, which also enter design mode the moment they are dragged onto either the visual or nonvisual design surface. The nonvisual area, or component tray, hosts components. A component is a Framework class that supports the design-time experience but doesn't draw its own UI in a container-specified region. Components implement the System.ComponentModel.IComponent interface, typically by deriving from the SystemComponent.Model.Component base class. Controls, on the other hand, do draw themselves and are therefore shown in the visual area. All controls ultimately derive from the System.Windows.Forms.Control class, which in turn derives from the Component base class. In other words, a control is really a component with a UI. Both act nearly identically at design time in terms of their integration with the design host. Figure 1 shows a form in design mode acting as a design host for a number of components and controls.

Figure 1 Form in Design Mode
Figure 1 Form in Design Mode

While both components and controls ultimately derive from the Component base class, that's not what distinguishes a component from any other class. Instead, what makes the Component class fit into the design-time architecture is its implementation of the IComponent interface. IComponent allows components to integrate with the design-time host, providing them with design-time services.

Hosts and Containers

In Visual Studio® .NET, the Windows Forms Designer is responsible for providing design-time services during Windows Forms development, including a form's UI and code views. The responsibility of managing integration between design-time objects and the designer is handled by an internal implementation of System.ComponentModel.Design.IDesignerHost. The designer host stores IComponent references to all design-time objects on the current form as well as the form itself, which is also a component. This collection of components is available from the IDesignerHost interface through the Container property of type System.ComponentModel.IContainer:

interface IContainer : IDisposable {
  ComponentCollection Components { get; }
  void Add(IComponent component);
  void Add(IComponent component, string
      name);
  void Remove(IComponent component);
}
This implementation of IContainer allows the designer host to establish a relationship that helps it manage each of the components placed on the form.

Sites

At design time, contained components can access the designer host, as well as each other, through their container. This two-way relationship is shown in Figure 2. You can also see that the fundamental relationship between the designer host and its components is established with an implementation of the System.ComponentModel.ISite interface:

interface ISite : IServiceProvider {
  IComponent Component { get; }
  IContainer Container { get; }
  bool DesignMode { get; }
  string Name { get; set; }
}

Figure 2 Designer Component Architecture
Figure 2 Designer Component Architecture

Internally, a container stores an array of sites. When each component is added to the container, the designer host creates a new site, connecting the component to its design-time container and vice versa, by caching the ISite interface in the IComponent.Site property implementation.

interface IComponent : IDisposable {
  ISite Site { get; set; }
  event EventHandler Disposed;
}

The Component base class implements IComponent and caches the site's interface in a property. It also provides a helper property to go directly to the component's container without having to go through the site first:

class Component : MarshalByRefObject, IComponent, IDisposable {
  public IContainer Container { get; }
  public ISite Site { virtual get; virtual set; }
}

The Component base class provides a component with access to both the container and the site directly. A component can also access the Visual Studio .NET designer host by requesting the IDesignerHost interface from the container:

IDesignerHost designerHost = this.Container as
    IDesignerHost;

In Visual Studio .NET, the designer has its own implementation of the IDesignerHost interface, but to fit into other designer hosts, it's best for a component to rely only on the interface and not any specific implementation.

Testing a Component

Here's the skeleton of our basic digital/analog clock control sample, including an Alarm property and a SoundTheAlarm event:

class ClockControl : Control {
  public ClockControl() { ... }
  public DateTime Alarm { get { ... } set { ... } }
  public event onSoundTheAlarm SoundTheAlarm;
  •••
}
Figure 3 shows the clock control in action.

Figure 3 Clock Control
Figure 3 Clock Control

Testing a component at design time requires a little more attention since you're likely to be testing in the same environment that you're developing in, namely Visual Studio .NET. Things you might test include adding it to the Toolbox, dragging it onto a form, configuring properties, consuming events, and execution. The simplest way to support all of this is to create a host Windows Application project, adding it to the same solution as your control project (see Figure 4).

Figure 4 Create a Host
Figure 4 Create a Host

Customizing the Toolbox

If your component is a User Control (and therefore derives from System.Windows.Forms.UserControl), you'll find that it automatically appears in the Toolbox under the Windows Forms tab after a successful build. For other components, you'll need to add them to the Toolbox manually. First, right-click the Toolbox and select Customize Toolbox. Then select the .NET Framework Components tab. Click the Browse button, navigate to the location of your control's assembly file, and click the Open button. Then click OK.

Once the component is firmly entrenched on the Toolbox, you can indulge in an Alan Cooper-inspired drag-and-drop orgy, spreading your components into Windows Forms projects far and wide.

Debugging

In a standard Windows Forms application, debugging is a matter of setting a breakpoint in some of your component's code and running the test application, relying on Visual Studio .NET to break at the right moment. Debugging design-time components requires a similar approach in that the DLL is going to need a hosting EXE to debug against. What makes testing design-time functionality a bit different is that you need a design-time host to debug against, and the favorite one is Visual Studio .NET itself. This means that you'll be using one instance of Visual Studio .NET with your component's project loaded to debug another instance of Visual Studio .NET with an instance of the component itself loaded. While this may seem somewhat confusing, it's remarkably easy to set up:

  1. Open the component to debug into one instance of Visual Studio .NET.
  2. Set a second instance of Visual Studio .NET as your debug application by going to Project | Properties | Configuration Properties | Debugging. Set Debug Mode to Program. Set Start Application to Devenv.exe path\devenv.exe. Then set Command Line Arguments to Test Solution Path\TestSolution.sln.
  3. Select Set as StartUp Project on your component project.
  4. Set a breakpoint in the component.
  5. Use Debug | Start (or F5) to begin debugging.
At this point, a second instance of Visual Studio .NET starts up with another solution (don't attempt to load the solution for the component you're testing into the second instance of Visual Studio .NET) and you can debug at will.

Components versus Controls

Although components and controls are similar as far as their design-time interaction is concerned, they are not identical. The most obvious difference is how they are drawn on the design surface. One not-so-obvious divergence is what happens when you drag one onto the design surface. When dropping a control, a design-time instance is created to support various design-time features, including the generation of initialization code to set the properties of the control (see Figure 5).

Figure 5 Designer-managed InitializeComponent (Close Figure 5)
class ClockControlHostForm : Form {
  ClockControlLibrary.ClockControl clockControl1;
  •••
  void InitializeComponent() {
    this.clockControl1 = new ClockControlLibrary.ClockControl();
    •••
    // clockControl1
    this.clockControl1.Alarm =
        new System.DateTime(2002, 11, 26, 3, 39, 42, 967);
    this.clockControl1.Location =
        new System.Drawing.Point(18, 16);
    this.clockControl1.Name = "clockControl1";
    this.clockControl1.Size = new System.Drawing.Size(256, 238);
    this.clockControl1.TabIndex = 0;
    this.clockControl1.Text = "clockControl1";
    •••
  }
  •••
}

InitializeComponent is a designer-managed area of your form code to which the designer serializes relevant initialization code for the controls and components hosted on the form. The designer uses InitializeComponent to remember control state across form loads at design time and to initialize controls for run-time execution. The designer does not generate the same code for every type of object that can be dragged onto a form. In particular, components that aren't also controls get all of the same code for creation and initialization as controls do, but they also get some extra code that notifies them when their container goes away. Controls don't need this extra code because they already get the Closed event, which is an equivalent notification for most purposes. A component can let the designer know that it would like to be notified when its container goes away by implementing a public constructor that takes a single argument of type IContainer, as shown in this snippet:

class ClockComponent : Component {
  public ClockComponent(IContainer container) {
    // Add object to container's list so that
    // we get notified when the container goes away
    container.Add(this);
  }
  •••
}

Notice that the constructor uses the container to add itself as a container component. In the presence of this constructor, the designer will generate code that uses this constructor, passing it a container for the component to add itself to. For example, if our clock sample were a component instead of a control, once it were dragged onto a design surface, the initialization code would call the special constructor:

class ClockComponentHostForm : Form {
  ClockComponent clockComponent1;
  IContainer components;
  •••
  void InitializeComponent() {
    this.components = new System.ComponentModel.Container();
    this.clockComponent1 = new ClockComponent(this.components);
    •••
  }
  •••
}

By default, the generated code for anything with a design surface will notify each component in the container as part of the Dispose method implementation:

class ClockComponentHostForm : Form {
  •••
  protected override void Dispose( bool disposing ) {
    if( disposing ) {
      if (components != null) {
        // Call each component's Dispose method
        components.Dispose();
      }
    }
    base.Dispose( disposing );
  }
}
A component that has added itself to the container can catch the notification that is being disposed by overriding the Component base class's Dispose method:
class ClockComponent : Component {
  •••
  protected override void Dispose(bool disposing) {
    // Dispose of resources
    •••

    // Let base class have a chance
    base.Dispose(disposing);
  }
}
This service of the designer allows all types of components to be immediately notified when the container goes away, although this happens differently depending on whether the component is actually a component or a control.

Design Mode

As soon as the basic clock control is hosted at design time, it's clear that it is a bit overzealous in keeping time:

   void timer_Tick(object sender, EventArgs e) {
     this.Refresh();
     •••
   }
The code contained in this timer event implements run-time functionality that shouldn't be executed at design time. To avoid executing such code, a control can check whether it's currently in design-time or run-time mode with Component.DesignMode, as shown here:
void timer_Tick(object sender, EventArgs e) {
  if( this.DesignMode ) return;
  this.Refresh();
  •••
}

Note that the DesignMode property shouldn't be checked from within the constructor, or any code called by the constructor. Since the constructor is called prior to the component being assigned to a site, and it's the site that determines whether a component is in design mode or not, an object can't know whether it's in design mode until after the constructor has finished executing.

Attributes

Design-time functionality is available to controls in one of two ways: programmatically or declaratively. DesignMode is an example of the programmatic approach. One side effect of using a programmatic approach is that your component implementation takes on design-time responsibilities, resulting in a blend of design-time and run-time code. On the other hand, a declarative approach relies on attributes to request design-time functionality implemented somewhere else, most often in the designer host itself. As an example attribute, consider the default Toolbox icon for a component, which you can see in Figure 6.

Figure 6 Old Icon
Figure 6 Old Icon

Using System.Drawing. ToolboxBitmapAttribute, you can give your component a professional look by specifying a more appropriate image. The first thing to do is add a 16 × 16 pixel, 16-color icon or bitmap to your project and set its Build Action to Embedded Resource. Then, to associate it with your component, add the ToolboxBitmapAttribute:

[ ToolboxBitmapAttribute(typeof(ClockControl),
  "images.ClockControl.ico") ]
class ClockControl : Control { ... }
The parameters to this attribute specify the use of an icon resource located in the "images" project subfolder.

You'll find that the Toolbox image doesn't change if you add or change ToolboxBitmapAttribute after the component has been added to the Toolbox. However, if your implementation is just a component (and not a control), its icon is updated in the component tray. You can only assume that the Toolbox is not under the direct management of the Windows Forms designer, unlike the component tray. To refresh the Toolbox, remove and then re-add the component to the Toolbox. The result will be something like Figure 7.

Figure 7 New Icon
Figure 7 New Icon

You can achieve the same result without using ToolboxBitmapAttribute by placing a 16 × 16 pixel, 16-color bitmap into the same project folder as the control and giving it the same name as the control class. This is a special case shortcut for the ToolboxBitmapAttribute only, so don't expect to find similar shortcuts for other design-time attributes.

Property Browser

Aside from code serialization, another interesting thing the designer does with your component is make it accessible through the property browser. The designer uses reflection with the design-time control instance to discover its public properties. It then calls the get accessor on each property to find the current value, rendering the property names and associated values in the property browser. Figure 8 shows how the property browser looks for the basic clock control.

Figure 8 Clock Properties
Figure 8 Clock Properties

The System.ComponentModel namespace provides a comprehensive set of attributes (see Figure 9) to help your control modify its appearance in the property browser. By default, public read and read/write properties are displayed in the property browser under the "Misc" category, such as the Alarm property which is highlighted in Figure 10.

Figure 9 Design-time Property Browser Attributes (Close Figure 9)

Attribute | Description
Browsable | Determines whether the property is visible in the property browser
Category | Tells the property browser in which group to include this property
Description | Provides text for the property browser displays in its description bar
DesignOnly | Specifies that the design-time value of this property is serialized to the form's resource file, and is typically used on design-time-only properties
MergableProperty | Allows this property to be combined with properties from other objects when more than one are selected and edited
ParenthesizePropertyName | Specifies whether this property should be surrounded by parentheses in the property browser
ReadOnly | Specifies that this property cannot be edited in the property browser
Figure 10 Alarm Property
Figure 10 Alarm Property

If a property is intended for run time only, it can be hidden from the property browser by including the BrowsableAttribute with the property, like this:

[ BrowsableAttribute(false) ]
public bool IsItTimeForABreak {
  get { ... }
  set { ... }
}
With IsItTimeForABreak out of the design-time picture, only the custom Alarm property remains. However, it's currently listed under the property browser's Misc category and lacks a description. We can improve the situation by applying both the CategoryAttribute and the DescriptionAttribute, like so:
[
  CategoryAttribute("Behavior"),
  DescriptionAttribute("Alarm for late risers")
]
public DateTime Alarm {
  get { ... }
  set { ... }
}
After adding these attributes and rebuilding, you'll notice that the Alarm property has relocated to the desired category in the property browser, and the description appears on the description bar when you select the property (both shown in Figure 11). You can actually use the CategoryAttribute to create new categories, but only do so if the existing categories don't describe the property; otherwise, you'll confuse users who are looking for your properties.

Figure 11 New Category
Figure 11 New Category

In Figure 11, you might have noticed that some property values are bold and some are not. Bold values are those that differ from the property's default value, as specified by the DefaultValueAttribute shown here:

[
  CategoryAttribute("Appearance"),
  DescriptionAttribute("Determines whether digital time is shown"),
  DefaultValueAttribute(true)
]
public bool ShowDigitalTime {
  get { ... }
  set { ... }
}
Using DefaultValueAttribute also allows you to reset a property to its default value via the property browser, which is available from the property's context.

The Reset option is disabled if the current property value is the default. Non-default properties are rendered in bold to provide a visual cue that they've been changed.

Like properties, a class can also have default values, most notably events and properties. By adorning a class with the DefaultEventAttribute, the designer will automatically hook up this event for you when the control is double-clicked:

[ DefaultEventAttribute("SoundTheAlarm") ]
class ClockControl : Control { ... }
Likewise, adorning the class with the DefaultPropertyAttribute will cause the property browser to highlight the property by default:
[ DefaultPropertyAttribute("ShowDigitalTime") ]
class ClockControl : Control { ... }

Code Serialization

Unlike the DefaultEventAttribute and DefaultPropertyAttribute, the DefaultValueAttribute serves a dual purpose. Not only does it affect a property's appearance in the property browser, but it also influences what code the designer serializes to InitializeComponent. Those with a default value are only included if the property's value is different than the default. The primary rule of thumb is that your initial property values should match the value set by the DefaultValueAttribute.

DesignerSerializationVisibilityAttribute is another attribute that affects the code serialization process. Its constructor takes a value from the DesignerSerializationVisibility enumeration. The default, Visible, causes a property's value to be set in InitializeComponent if its value is not the same as that of the default (as we've seen). If you'd prefer that no code is generated to initialize a property, no matter what the value, use Hidden:

[
  CategoryAttribute("Appearance"),
  DescriptionAttribute("Determines whether digital time is shown"),
  DefaultValueAttribute(true),
  DesignerSerializationVisibilityAttribute
  (DesignerSerializationVisibility.Hidden)
]
public bool ShowDigitalTime {
  get { ... }
  set { ... }
}
You should use Hidden in conjunction with the Browsable attribute set to false for run-time-only properties.

Batch Initialization

You might have noticed that the code that does eventually get serialized to InitializeComponent is laid out as an alphanumerically ordered sequence of property sets, grouped by object. Order isn't important until your component or control exposes range-dependent properties, such as Min/Max or Start/Stop pairs. The clock control also has two dependent properties, PrimaryAlarm and the BackupAlarm (we decided to split the Alarm property into two for extra-sleepy people).

Internally, the clock control instance initializes the two properties 10 minutes apart, starting from the current date/time:

DateTime _primaryAlarm = DateTime.Now;
DateTime _backupAlarm = DateTime.Now.AddMinutes(10);

Both properties should check to ensure the values are valid, as shown in Figure 12. Figure 13 shows the resulting code when these properties are serialized to InitializeComponent. At design time, the property browser would show an error dialog if you enter an invalid property into the property browser. That's OK, because it lets the developer know the relationship between the two properties. However, even if the developer sets them appropriately, we still have a problem at runtime. The problem is that properties are serialized in InitializeComponent in alphabetical order. When BackupAlarm is set, validation takes place resulting in an exception being thrown because the PrimaryAlarm property hasn't been set yet. What we'd like is to be notified when properties are being set in batch mode so that we can validate them all at once, no matter what order they're set in.

Figure 13 Designer-serialized Property Initialization (Close Figure 13)
void InitializeComponent() {
  •••
  //
  // clockControl1
  //
  this.clockControl1.BackupAlarm = new System.DateTime(2003, 1, 31, 6, 31,
      27, 685);
  •••
  this.clockControl1.PrimaryAlarm = new System.DateTime(2003, 1, 30, 0, 0,
      0, 0);
  •••
  }
Figure 12 Check for Valid Values (Close Figure 12)
public DateTime PrimaryAlarm {
  get { return _primaryAlarm; }
  set {
    if( value >= _backupAlarm )
      throw new Exception("Primary alarm must be before Backup alarm");
    _primaryAlarm = value;
  }
}
public DateTime BackupAlarm {
  get { return _backupAlarm; }
  set {
    if( value < _backupAlarm )
      throw new Exception("Backup alarm must be after Primary alarm");
    _backupAlarm = value;
  }
}

Implementing System.ComponentModel.ISupportInitialize provides this capability with two notification methods called before and after initialization:

public interface ISupportInitialize {
  public void BeginInit();
  public void EndInit();
}

When implemented, calls to BeginInit and EndInit are serialized to InitializeComponent (see Figure 14). Placing the appropriate validation logic into EndInit lets us delay property checking until after initialization, as shown here:

class ClockControl : ..., ISupportInitialize {
  void ISupportInitialize.BeginInit() {}
  void ISupportInitialize.EndInit() {
    // Check alarm values
    if( _primaryAlarm >= _backupAlarm )
      throw new Exception("Primary alarm must be before backup alarm");
  }
  •••
}

Figure 14 Initialization Notification (Close Figure 14)
void InitializeComponent() {
  •••
  ((System.ComponentModel.ISupportInitialize) (this.clockControl1)).BeginInit();
  •••
  // clockControl1
  this.clockControl1.BackupAlarm = new System.DateTime(2003, 1, 31, 6, 31,
      27, 685);
  •••
  this.clockControl1.PrimaryAlarm = new System.DateTime(2003, 1, 30, 0, 0,
      0, 0);
  •••
  ((System.ComponentModel.ISupportInitialize) (this.clockControl1)).EndInit();
  •••
}

The final step is to disable both PrimaryAlarm and BackupAlarm's property setter validation logic during initialization, with a simple flag representing the component's initialization state (see Figure 15).

Figure 15 Disabling Property Validation (Close Figure 15)
class ClockControl : ..., ISupportInitialize {
  •••
  private bool  _initializing = false;
  •••
  void ISupportInitialize.BeginInit() { _initializing = true; }
  void ISupportInitialize.EndInit() {
    if( !this.DesignMode ) { ••• }
    _initializing = false;
  }
  •••
  public DateTime PrimaryAlarm {
    get { ••• }
    set {
      if( !_initializing ) { ••• }
      _primaryAlarm = value;
    }
  }

  public DateTime BackupAlarm {
    get { ••• }
    set {
      if( !_initializing ) { ••• }
      _backupAlarm = value;
    }
  }
}

Now the control supports design time and run time validation checking, independent of property setting order in InitializeComponent.

EndInit also turns out to be a better place to avoid the timer's Tick event, which currently fires once every second during design time. Placing the timer's initialization logic into EndInit prevents the events from firing until run time:

class ClockControl : ..., ISupportInitialize {
  void ISupportInitialize.BeginInit() {}
  void ISupportInitialize.EndInit() {
    if( !this.DesignMode ) {
      •••
      //Initialize timer
      _timer.Interval = 1000;
      _timer.Tick += new System.EventHandler(this.timer_Tick);
      _timer.Enabled = true;
    }
  }
  •••
}

Extender Controls

So far the discussion has focused on the properties implemented by a control for itself, but there are situations where a control might actually provide properties to other controls. For example, the clock control only tells the time in the current time zone. Adding a TimeZoneModifier property would allow the clock control to be adjusted to tell the time in other time zones. This is an example of a property that might be better provided or extended to other controls.

One way to use this feature could be to display the time in each time zone in which your organization has offices. If each office was visually represented with a picture box, you could drag one clock control for each time zone onto the form, adjusting the TimeZoneModifier property on each clock control manually. The result might look like the clocks in Figure 16.

Figure 16 Time Zones
Figure 16 Time Zones

This works quite nicely, but could lead to space problems, particularly if you have one clock control for each of the 24 time zones globally and, consequently, there are 24 implementations of the same logic on the form. Figure 17 shows how this might look.

Figure 17 Too Many Instances
Figure 17 Too Many Instances

Another approach might be to have a single clock control and update its TimeZoneModifier property with the relevant time zone from, say, the Click event of each picture box. This is a more cumbersome approach because it requires developers to write the code associating a time zone offset with each control that affects the time zone, as shown in Figure 18.

Figure 18 One Clock Control
Figure 18 One Clock Control

It would be nice to be able to access a single implementation of the clock control without forcing the developer to write code to associated per-control property values for the clock. The .NET Framework provides extender properties to do just this.

Logically, an extender property is a property provided by an extender control (like the clock control) on other controls in the same container (like the picture boxes). By implementing the TimeZoneModifier as an extender property, each picture box control on the form can get its own value, as shown in Figure 19.

Figure 19 Better Way to Use One Control
Figure 19 Better Way to Use One Control

Exposing an extender property from your control first requires that you declare the property to extend using the ProvidePropertyAttribute, like this:

[ ProvidePropertyAttribute("TimeZoneModifier", typeof(PictureBox)) ]
public class ClockControl : System.Windows.Forms.Control { ... }
The first parameter to the attribute is the name of the property to extend. The second parameter is the "receiver" type, which specifies the type of object to extend, such as Control or Button. Only controls of the type specified by receiver can be extended. If you'd like to implement a more sophisticated algorithm, like supporting picture boxes and panels for example, you'll need to implement IExtenderProvider like so:
class ClockControl : ..., IExtenderProvider {
  bool IExtenderProvider.CanExtend(object extendee) {
    // Don't extend self
    if( extendee == this ) return false;

    // Extend suitable controls
    return( (extendee is PictureBox) ||
            (extendee is Panel) );
  }
  •••
}

As shown in Figure 20, the provider supports one or more extendee controls. Consequently, the provider control must be able to store and distinguish one extendee's property value from another. It does this with GetPropertyName and SetPropertyName methods, where PropertyName is the name you provided in the ProvidePropertyAttribute. GetTimeZoneModifier simply has to return the property value when requested by the property browser:

public class ClockControl : System.Windows.Forms.Control,
IExtenderProvider {
  private HashTable _timeZoneModifiers = new HashTable();

     public string GetTimeZoneModifier(Control extendee) {
       return Convert.ToString(_timeZoneModifiers[extendee]);
     }
     •••
   }

Figure 20 Extended TimeZoneModifier Property Setter (Close Figure 20)
class ClockControl : ..., IExtenderProvider {
  private HashTable _timeZoneModifiers = new HashTable();

  public void SetTimeZoneModifier(Control extendee, object value) {
    string timeZoneModifier = (string)value;
    // If property isn't provided
    if( (timeZoneModifier == "") || (timeZoneModifier == null) ) {
      // Remove it
      _timeZoneModifiers.Remove(extendee);
      if( !this.DesignMode ) { extendee.Click -= new
          EventHandler(extendee_Click); }
    }
    else {
      // Add it, convert it to Int32 to make sure it's the right type
      _timeZoneModifiers[extendee] = Convert.ToInt32(timeZoneModifier);
      if( !this.DesignMode ) { extendee.Click += new
          EventHandler(extendee_Click); }
    }
  }
  •••
}

SetTimeZoneModifier has the responsibility of storing and retrieving property values on behalf of the extendee control. Also, with the example TimeZoneModifier property, we need to hook into each extendee control's Click event unless the control isn't using the extended property. SetTimeZoneModifier is shown in Figure 20. As with normal properties, you can affect their appearance in the property browser by adorning the GetPropertyName property implementation with the usual suspects, which are shown here:

class ClockControl : ..., IExtenderProvider {
  [
    Category("Behavior"),
    Description("Sets the timezone difference from the current time"),
    DefaultValue("")
  ]
  public string GetTimeZoneModifier(Control extendee) { ... }
  •••
}

These attributes are applied to the extendee's property browser view. A compiled control's extended properties will appear in the extendee control's property browser as specified by the use or lack of attributes, with the following naming format:

<ExtendedPropertyName> on <ExtenderProviderName>
Figure 21 shows the extended TimeZoneModifier property in action, behaving like any other property, on a PictureBox control.

Figure 21 TimeZoneModifier
Figure 21 TimeZoneModifier

If a property is set and is not the default value, it is serialized to InitializeComponent grouped with the extendee control, but set on the extender control:

void InitializeComponent() {
  •••
  // pictureBox1
  this.pictureBox1.Location = new System.Drawing.Point(62, 163);
  this.pictureBox1.Name = "pictureBox1";
  this.pictureBox1.TabIndex = 5;
  this.pictureBox1.TabStop = false;
  this.clockControl1.SetTimeZoneModifier(this.pictureBox1, "-2");
  •••
}

Where Are We?

We've given you an overview of building and debugging controls that target the .NET Framework design-time features. You've also seen how attributes and interfaces enhance a control's relationship with the property browser and InitializeComponent. In a future article we'll show you some more advanced design-time features including TypeConverters, UITypeEditors, and Designers, which really spruce up your control's usability and flexibility.

For background information see:
Windows Forms by Chris Sells (to be published by Addison-Wesley in 2003)

Michael Weinhardtis a software engineer at SERF, a retirement fund company where he develops applications for the .NET Framework. He can be reached at mikedub@optusnet.com.au.

Chris Sellsis an independent consultant, speaker, and author specializing in distributed applications in the .NET Framework and COM. More information about Chris and his various projects is available at http://www.sellsbrothers.com.


ASP.NET: Nine Options for Managing Persistent User State in Your ASP.NET Application

New information has been added to this article since publication.
Refer to the Editor's Update below.

ASP.NET

Nine Options for Managing Persistent User State in Your ASP.NET Application

Steven A. Smith
This article assumes you're familiar with ASP.NET
Level of Difficulty 1 2 3
Code download available at: ASPNETUserState.exe (117KB)
SUMMARY

ASP.NET provides many different ways to persist data between user requests. You can use the Application object, cookies, hidden fields, the Session or Cache objects, and lots of other methods. Deciding when to use each of these can sometimes be difficult. This article will introduce the aforementioned techniques and present some guidelines on when to use them. Although many of these techniques existed in classic ASP, best practices for when to use them have changed with the introduction of the .NET Framework. To persist data in ASP.NET, you'll have to adjust what you learned previously about handling state in ASP.

Contents
Application
Cookies
Form Post / Hidden Form Field
QueryString
Session
New State Containers in ASP.NET
Cache
Context
ViewState
Web.config and Machine.config Files
Conclusion

S
ince the dawn of the Web, managing state in the stateless world of HTTP has been a problem for Web developers. More recently, various techniques for storing and retrieving data have emerged. In this article I will describe how ASP.NET developers can maintain or pass state across page requests.

In ASP.NET, there are several different ways to persist data between user requests—so many, in fact, that the novice developer is often confused about which object to use in a particular situation. In order to answer this question, there are three criteria that should be considered:

  • Who needs the data?
  • How long does the data need to be persisted?
  • How large is the set of data?

By answering these questions, you can determine which object provides the best solution for persisting data between requests in an ASP.NET application. Figure 1 lists the different state management objects and describes when they should be used. In ASP.NET, four new objects have been added: Cache, Context, ViewState, and the Web.Config file. Classic ASP objects that are also available in ASP.NET include Application, Cookie, Form Post with a hidden form field, QueryString, and Session. Note that the proper use of these five data containers has changed, so experienced ASP programmers may have some "unlearning" to do when considering these familiar objects.

Figure 1 Data Container Objects in ASP.NET (Close Figure 1)

Persistence Method | Who Needs the Data? | For How Long? | How Much Data?
Application | All users | Until the next application restart | Can be almost any size—it will only be stored once
Cookie | One user | As short as desired, or for months or even years if the user doesn't delete their cookies | Minimal, simple data
Form Post | One user | For the next request (can be reused across many requests) | Virtually any size—the data is sent back and forth with every page
QueryString | One user or one group of users | For the next request (can be reused across many requests) | Minimal, simple data
Session | One user | As long as the user is active, plus a timeout period (typically 20 minutes) | Can be almost any size, but should be minimized since every user has their own separate session store
Cache | All users or a subset of users | As long or as short as needed | Can be used for large or small, simple or complex data
Context | One user | This request only | Can hold large objects, but typically does not since it is often used for every request
ViewState | One user | One Web form | Minimal; as with Form Post, this data is sent back and forth with every page
Config file | All users | Until the configuration file is updated | Can hold a lot of data; usually organized as many small strings or XML structures

Application

Let's set the object use criteria by answering the state questions I asked earlier. Who needs this data? All users need access to it. How long does this data need to be persisted? It has to live forever, or for the life of the application. How large is this data? It can be almost any size—only one copy of the data will exist at any given time.

In classic ASP, the Application object provided a great place to store frequently used pieces of data that changed infrequently, such as the contents of menus or other reference data. While the Application object is still available as a data container in ASP.NET, other objects are generally better suited for the kinds of data that would have been stored in the Application collection of a classic ASP application.

In classic ASP, the Application object was an ideal choice if the data that was to be stored did not vary at all (or very rarely) for the life of the application (such as read-only or read-mostly data). Connection strings were one of the more common pieces of data stored in Application variables, but in ASP.NET such configuration data is best stored in the Web.config file. One thing to consider if you are using the Application object is that any writes to it should be done either in its Application_OnStart event (in global.asax) or within an Application.Lock section. While using Application.Lock is necessary to ensure that writes are performed properly, it also serializes requests for the Application object, which can be a serious performance bottleneck for the application. Figure 2 demonstrates how to use the Application object; it consists of a Web form and its code-behind file. An example of its output is shown in Figure 3.

Figure 2 Accessing the Application Object in ASP.NET (Close Figure 2)

Application.aspx
<form id="Application" method="post" runat="server">
    <asp:validationsummary id="valSummary" Runat="server">
        </asp:validationsummary>
    <table>
        <tr>
            <td colSpan="3">Set Application Variable:</td>
        </tr>
        <tr>
            <td>Name</td>
            <td><asp:textbox id="txtName" Runat="server"></asp:textbox>
                </td>
            <td><asp:requiredfieldvalidator id="nameRequired"
                 runat="server" Display="Dynamic" ErrorMessage="Name is
                 required." ControlToValidate="txtName">*
                 </asp:requiredfieldvalidator></td>
        </tr>
        <tr>
            <td>Value</td>
            <td><asp:textbox id="txtValue" Runat="server">
                </asp:textbox></td>
            <td><asp:requiredfieldvalidator id="valueRequired"
                 Runat="server" Display="Dynamic" ErrorMessage="Value is
                 required." ControlToValidate="txtValue">*
                 </asp:requiredfieldvalidator></td>
        </tr>
        <tr>
            <td colSpan="3"><asp:button id="btnSubmit" Runat="server"
                             Text="Update Value"></asp:button></td>
        </tr>
    </table>
    <asp:Label ID="lblResult" Runat="server" />
</form>
Application.aspx.cs
private void btnSubmit_Click(object sender, System.EventArgs e)
{
    if(IsValid)
    {
        Application.Lock();
        Application[txtName.Text] = txtValue.Text;
        Application.UnLock();

        lblResult.Text = "The value of <b>" + txtName.Text +
            "</b> in the Application object is <b>" +
            Application[txtName.Text].ToString() + "</b>";
    }
}
Figure 3 Contents of Application Object
Figure 3 Contents of Application Object

Note that in Figure 3 the contents of the Application object are displayed in the trace output. Tracing is a really great debugging tool, but you can expect that at some point, a page with tracing turned on will accidentally make it into your production environment. When that happens, you really won't want anything sensitive to be shown. This is one of the primary reasons why the Application object is no longer the recommended place to store sensitive information like connection strings.

Cookies

Cookies are handy when a particular user needs a specific piece of data, and it needs to be persisted for a variable period of time. It can be as brief as the life of the browser window, or as long as months or even years. As far as size goes, cookies are very small. Cookies can be as small as only a few bytes of data, and since they are passed with every browser request, their contents should be kept as small as possible.

Cookies provide a very flexible and powerful way to maintain data between user requests, which is why most dynamic sites on the Internet use them. Since the amount of data cookies can store is very limited, it is best to just store key fields in the cookie, with the rest of the data kept in a database or other server-side data container. However, because cookies are not supported by all browsers and can be disabled or deleted by users, they should never be used for critical data. You should also be sure to always gracefully handle the case in which a user's cookie has been deleted. Finally, cookies are stored as simple plain text on the user's machine; therefore sensitive, unencrypted data should not be stored in them. [ Editor's Update - 1/11/2005: The best way to secure sensitive state that should not be viewed or modified by a hostile user is to store that state on the server. If sensitive data must be sent to the client, it should be encrypted beforehand, regardless of the storage mechanism employed.]

A particular named cookie can store a single value or a collection of name/value pairs. Figure 4 shows an example of both single- and multi-value cookies, as output by the built-in trace features of ASP.NET. These values can be manipulated within an ASP.NET page by using the Request.Cookies and Response.Cookies collections, as Figure 5 demonstrates.

Figure 5 Accessing Cookies in ASP.NET (Close Figure 5)

Cookies.aspx.cs
// Setting a cookie's value and/or subvalue using the HttpCookie class
HttpCookie cookie;
if(Request.Cookies[txtName.Text] == null)
    cookie = new HttpCookie(txtName.Text, txtValue.Text);
else
    cookie = Request.Cookies[txtName.Text];
if(txtSubValueName.Text.Length > 0)
    cookie.Values.Add(txtSubValueName.Text, txtSubValueValue.Text);
cookie.Expires = System.DateTime.Now.AddDays(1); // tomorrow
Response.AppendCookie(cookie);

// Retrieving a cookie's value(s)
if(!Request.Cookies[txtName.Text].HasKeys)
    lblResult.Text = "The value of the <b>" + txtName.Text + "</b>
        cookie is <b>" + Request.Cookies[txtName.Text].Value.ToString() +
        "</b>";
else
{
    lblResult.Text = "The value of the <b>" + txtName.Text + "</b>
        cookie is <b>" + Request.Cookies[txtName.Text].Value.ToString() +
        "</b>, with subvalues:<br>";
    foreach(string key in Request.Cookies[txtName.Text].Values.Keys)
    {
        lblResult.Text += "[" + key + " = " +
            Request.Cookies[txtName.Text].Values[key].ToString() + "]<br>";
    }
}
Delete a Cookie
// Set the value of the cookie to null and
// set its expiration to some time in the past
Response.Cookies[txtName.Text].Value = null;
Response.Cookies[txtName.Text].Expires =
    System.DateTime.Now.AddMonths(-1); // last month
Figure 4 Single-value and Multi-value Cookies (Close Figure 4)

Form Post / Hidden Form Field

Form data is needed by a particular user, and it must be persisted for any period from a single request to the life of the application. The data can be virtually any size; it's sent back and forth over the network with each form post.

In classic ASP, this was a common way to retain state within an application, especially in a multi-page form. However, in ASP.NET, this technique is rarely appropriate since Web controls and ViewState handle this automatically as long as you use the postback model (that is, pages that post back to themselves). ViewState is an ASP.NET implementation of this technique, which I will describe later in this article. Access to form values sent through a POST is done using the HttpRequest object's Form collection. In Figure 6, a user ID is set by one ASP.NET page, after which it is persisted in a hidden form field. Subsequent requests to either page retain the value as long as the pages use Submit buttons to link to each other.

Figure 6 Using Hidden Form Fields to Persist Data in ASP.NET (Close Figure 6)

Form1.aspx
<h1>Form 1</h1>
<form id="Application" method="post" runat="server">
    <p>Your username:
<asp:Label ID="lblUsername" Runat="server" />
</p>
    <asp:Panel Runat="server" ID="pnlSetValue">
    <asp:validationsummary id="valSummary" Runat="server">
        </asp:validationsummary>
    <TABLE>
    <TR>
        <TD colSpan="3">Set Hidden Form Username Variable:</TD></TR>
    <TR>
        <TD>Username</TD>
        <TD>
    <asp:textbox id="txtName" Runat="server"></asp:textbox></TD>
        <TD>
    <asp:requiredfieldvalidator id="nameRequired" runat="server"
        ControlToValidate="txtName" ErrorMessage="Name is required."
        Display="Dynamic">*</asp:requiredfieldvalidator></TD></TR>
    <TR>
        <TD colSpan="3">
    <asp:button id="btnSubmit" Runat="server" Text="Set Value">
        </asp:button></TD></TR></TABLE>
    </asp:Panel>
    <asp:Label ID="lblResult" Runat="server" />
</form>
<form action="form2.aspx" method="post" name="form2" id="form2">
    <input type="hidden" name="username" value="<%# username %>" >
    <input type="submit" value="Go to Form2.aspx"
</form>
Form1.aspx.cs
private void Page_Load(object sender, System.EventArgs e)
{
    if(!IsPostBack) // new request or request from form2.aspx
    {
        // check Forms collection
        if(Request.Form["username"] == null)
            pnlSetValue.Visible = true;
        else
        {
            // need to set the username value
            pnlSetValue.Visible = false;
            username = Request.Form["username"].ToString();
            lblUsername.Text = username;
            // Databind to set the hidden form field's value
            this.DataBind();
        }
    }
}

private void btnSubmit_Click(object sender, System.EventArgs e)
{
    if(IsValid)
    {
        // Hide the form to set the value.
        pnlSetValue.Visible = false;
        username = txtName.Text;
        lblResult.Text = "Username set to " + txtName.Text + ".";
        lblUsername.Text = username;
        this.DataBind();
    }
}
Form2.aspx
<h1>Form 2</h1>
<form id="Application" method="post" runat="server">
    <p>Your username: <asp:Label ID="lblUsername" Runat="server" /></p>
</form>
<form action="form1.aspx" method="post" id="form2" name="form2">
    <input type="hidden" name="username" value="<%# username %>" >
    <input type="submit" value="Go to Form1.aspx"
</form>
Form2.aspx.cs
private void Page_Load(object sender, System.EventArgs e)
{
    if(Request.Form["username"] != null)
    {
        username = Request.Form["username"].ToString();
        lblUsername.Text = username;
        this.DataBind();
    }
}

In ASP.NET, only one server-side form can exist on a page, and that form must submit back to itself (client-side forms can still be used, without limitations). One of the major reasons that hidden form fields are no longer used to pass data around applications built on the Microsoft® .NET Framework is that all .NET Framework controls are capable of maintaining their own state automatically using ViewState. ViewState simply encapsulates the work involved in setting and retrieving values using hidden form fields into a simple-to-use collection object.

QueryString

The data stored in the QueryString object is used by the individual user. Its lifetime can be as brief as a single request, or as long as the user continues to use the application (if architected appropriately). This data is typically less than 1KB. Data in a QueryString is passed in the URL and is visible to the user, so as you might guess, sensitive data or data that can be used to control the application should be encrypted when using this technique.

That said, the QueryString is a great way to send information between Web forms in ASP.NET. For example, if you have a DataGrid with a list of products, and a hyperlink in the grid that goes to a product detail page, it would be an ideal use of the QueryString to include the product ID in the QueryString of the link to the product details page (for example, productdetails.aspx?id=4). Another advantage of using QueryStrings is that the state of the page is contained in the URL. This means that a user can put a page in their Favorites folder in its generated form when it's created with a QueryString. When they return to it as a favorite, it will be the same as when they actually made it a favorite. Obviously, this only works if the page doesn't rely on any state outside the QueryString and nothing else changes.

Along with sensitive data, any variable that you don't want the user to be able to manipulate should be avoided here (unless encryption is used to remove human-readability). Also, keep in mind that characters that are not valid in a URL must be encoded using Server.UrlEncode, as Figure 7 shows. When dealing with a single ASP.NET page, ViewState is a better choice than QueryString for maintaining state. For long-term data storage, Cookie, Session, or Cache are more appropriate data containers than QueryStrings.

Figure 7 Using QueryStrings to Pass Data in ASP.NET (Close Figure 7)

Querystring.aspx
<form id="Querystring" method="post" runat="server">
    <asp:validationsummary id="valSummary" Runat="server">
        </asp:validationsummary>
    <table>
        <tr>
            <td colSpan="3">Set Querystring Variable:</td>
        </tr>
        <tr>
            <td>Name</td>
            <td><asp:textbox id="txtName" Runat="server"></asp:textbox>
                </td>
            <td><asp:requiredfieldvalidator id="nameRequired"
                 runat="server" Display="Dynamic" ErrorMessage="Name is
                 required." ControlToValidate="txtName">*
                 </asp:requiredfieldvalidator></td>
        </tr>
        <tr>
            <td>Value</td>
            <td><asp:textbox id="txtValue" Runat="server">
                    </asp:textbox></td>
            <td><asp:requiredfieldvalidator id="valueRequired"
                 Runat="server" Display="Dynamic" ErrorMessage="Value is
                 required." ControlToValidate="txtValue">*
                 </asp:requiredfieldvalidator></td>
        </tr>
        <tr>
            <td colSpan="3"><asp:button id="btnSubmit" Runat="server"
                             Text="Update Value"></asp:button></td>
        </tr>
    </table>
    <asp:Label ID="lblResult" Runat="server" />
    <a href="http://querystring.aspx?x=1">Set querystring x equal to 1</a>
</form>
Querystring.aspx.cs
private void Page_Load(object sender, System.EventArgs e)
{
    // Retrieving a cookie's value(s)
    if(Request.QueryString.HasKeys())
    {
        lblResult.Text = "The values of the <b>" + txtName.Text +
                         "</b> querystring parameter are:<br>";
        foreach(string key in Request.QueryString.Keys)
        {
            lblResult.Text += "[" + key + " = " +
                Request.QueryString[key].ToString() + "]<br>";
        }
    }
}

private void btnSubmit_Click(object sender, System.EventArgs e)
{
    if(IsValid)
    {
        string url = "querystring.aspx?";
        foreach(string key in Request.QueryString.Keys)
        {
            url += key + "=" + Request.QueryString[key].ToString() + "&";
        }
        Response.Redirect(url + txtName.Text + "=" +
            Server.UrlEncode(txtValue.Text));
    }
}

Session

Session data is specific to a particular user. It lives for as long as the user continues to makes requests plus some period of time afterward (typically 20 minutes). The Session object can hold large or small amounts of data, but total storage should be kept minimal if the application is intended to scale to hundreds of users or more.

Unfortunately, the Session object earned itself a very bad name in classic ASP because it tied an application to a particular machine, precluding the use of clustering and Web farms for scalability. In ASP.NET, this is less of an issue, since it is a simple matter to change the location where the session is stored. By default (and for best performance), session data is still stored in the memory of the local Web server, but ASP.NET also supports an external state server or database for managing Session data.

Using the Session object is easy and its syntax is identical to classic ASP. However, the Session object is one of the less efficient ways of storing user data, since it is held in memory for some time even after the user has stopped using the application. This can have serious effects on scalability for a very busy site. Other options allow more control over the release of memory, such as the Cache object, which may be better suited for some large data values. Also, ASP.NET sessions rely on cookies by default so if the user disables or doesn't support cookies, sessions won't work. Support for cookie-free sessions can be configured, however. For small amounts of data, the Session object can be a perfectly valid place to store user-specific data that needs to persist only for the duration of the user's current session. The following example demonstrates how to set and retrieve values from the Session object:

private void btnSubmit_Click(object sender, System.EventArgs e)
{
    if(IsValid)
    {
        // Set the Session value.
        Session[txtName.Text] = txtValue.Text;

        // Read and display the value we just set
        lblResult.Text = "The value of <b>" + txtName.Text +
            "</b> in the Session object is <b>" +
            Session[txtName.Text].ToString() + "</b>";
    }
}
The Web form is almost identical to the one used for the Application object, and the contents of the Session collection are also visible when page tracing is enabled.

You should be aware that even when not in use, sessions carry some overhead for an application. You can squeeze a little bit more performance out of your pages if you turn off sessions on pages that do not use it. Also, setting session state to read-only can also optimize pages that read but do not write data to sessions. Configure sessions in this fashion by adding an attribute to the @Page directive in one of these two ways::

<%@ Page EnableSessionState="false" %>
<%@ Page EnableSessionState="readonly" %>

ASP.NET sessions can be configured in the Web.config or Machine.config with the sessionState element. This element supports the attributes listed in Figure 8.

Figure 8 sessionState Attributes (Close Figure 8)

Attribute | Options | Description
mode | Off | Disables sessions.
 | Inproc | Same as classic ASP. Stored in the Web server's local memory. This option provides the best performance, but it is not clusterable. This is the default option.
 | StateServer | Session data is stored in memory on another server.
 | SqlServer | Session data is stored in a SQL Server database.
cookieless | True | Enables cookieless sessions. Session ID is automatically passed in the QueryString instead of in a cookie for all relative URLs in the application.
 | False | This is the default setting. Sessions use cookies.
timeout |  | Session timeout in minutes. The default is 20.
sqlConnectionString |  | Connection string used for SqlServer mode sessions.
stateConnectionString |  | Connection string used for StateServer mode sessions.

Here is an example of the settings in the Web.config:

<sessionState timeout="10" cookieless="false" mode="Inproc" />

New State Containers in ASP.NET

As mentioned earlier, ASP.NET adds several new ways to store data during and between user requests. This gives you much finer control over how state information is persisted. These new techniques narrow the scope down to as small as a single request (the Context object), or widen the scope to as large as the whole Web server and all applications on that server (the Machine.config file). In many cases, you will have several options available to you when you need to store a particular piece of data—use the questions and answers provided with each method description to determine if it is appropriate for your situation.

Cache

Cache data is specific to the single user, a subset of users, or even all users. This data persists for multiple requests. It can persist for a long time, but not across application restarts. Also, data can expire based on time or other dependencies. It can hold both large and small amounts of data effectively.

The Cache is one of the coolest objects in all of ASP.NET. It offers incredible flexibility, versatility, and performance, and is therefore often a better choice than Application or Session for persisting data within an ASP.NET application. A complete description of the ways in which the Cache object can be used (both declaratively and programmatically) is beyond the scope of this article, but suffice to say, it's a versatile object. Like the other collection objects, it is simply a name-value collection, but by using a key value that is specific to a user, you can make the cached values user-specific. Similarly, you can cache multiple sets of data for different related data, like several sets of car data with keys like "fordcars", "chevycars", and "gmcars". Data in the Cache can be given an expiration period that is absolute, sliding, or based on changes to a file. They also implement a callback function that is invoked whenever the cached value is ejected from the cache, which is useful because you can then check to see if there is a more recent version of the data available, and if not (or if the data source is unavailable), re-cache the value that was just expired.

Adding and accessing data in the cache is done using a syntax similar to what I have already covered. However, in addition to the standard indexer method of accessing this collection's contents, Cache also supports a number of methods to allow more control over the data that is cached. The method you will most often use is Insert, which supports several overloads that allow you to specify dependencies, timeouts, priority, and callbacks. Some simple examples are shown in the following code:

// Add item to cache
Cache["myKey"] = myValue;

// Read item from cache
Response.Write(Cache["myKey"]);

// Set a CacheDuration of 10 seconds and add item to cache
Cache.Insert("myKey",myValue, null, System.DateTime.Now.AddSeconds(10),
System.Web.Caching.Cache.NoSlidingExpiration);

One of the more powerful features of the Cache object is its ability to execute a callback when an item in the cache expires. This uses delegates or function pointers, a fairly advanced topic that I won't be covering in this article. Fortunately, once you have a bit of sample code showing how this technique works, you can take advantage of it in your applications by simply cutting and pasting, without knowing all the intricacies of how delegates work. There are many reasons why you might use this functionality, the most common being to refill the cache with current data whenever it expires, or restoring the old cache data if the data source to repopulate the cache is unavailable.

In my example, I am simply going to cache the current time, and whenever the cache expires, I am going to add an asterisk character (*) to the end of the string in the cache. Over time, you will be able to determine how many times the cache has expired by counting the asterisks. Figure 9 demonstrates the important concept of callbacks and provides a good template for building more functional callback routines into your use of the cache.

Figure 9 Caching Callback Example (Close Figure 9)
private void Page_Load(object sender, System.EventArgs e)
{
    string cacheKey = "myKey";
    string data = "";
    // Check to see if the data is in the cache already
    if(Cache[cacheKey]==null)
    {
        // Get the data since it isn't in the cache
        data = System.DateTime.Now.ToString();

        //create an instance of the callback delegate
        CacheItemRemovedCallback callBack =
            new CacheItemRemovedCallback(onRemove);

        Label1.Text = "Generated: " + data;

        Cache.Insert(cacheKey,data,null,
            System.DateTime.Now.AddSeconds(5),
            System.Web.Caching.Cache.NoSlidingExpiration,
            System.Web.Caching.CacheItemPriority.Default,
            callBack);
        }
    else
    {
        Label1.Text = "Cached: " + Cache[cacheKey].ToString();
    }
}

private void onRemove(string key, object val,
    CacheItemRemovedReason reason)
{
    //create an instance of the callback delegate
    CacheItemRemovedCallback callBack =
        new CacheItemRemovedCallback(onRemove);
    Cache.Insert(key,val.ToString() +
        "*",null,System.DateTime.Now.AddSeconds(5),Cache.NoSlidingExpiration,
        System.Web.Caching.CacheItemPriority.Default, callBack);
}

One important feature to note in Figure 9 is the pattern used in the Page_Load to determine whether or not to use the data in the cache. You will always want to use this pattern when you deal with items in the cache. Use an if statement to check if the current contents of the cache are null (use a variable for your cache key since you'll be referencing it several times). If it is null, generate the data from its source and place it in the cache. If it is not null, return the data from the cache. If you have some very complex data access logic, you should place the whole if statement in a separate function that's tasked with retrieving the data.

The Cache object has a lot more functionality than most of the other objects I have discussed. It is one of the more powerful features of ASP.NET, and I would definitely recommend reading more on it. The summary at the beginning of this article lists some places to start your search for more information.

Context

The Context object holds data for a single user, for a single request, and it is only persisted for the duration of the request. The Context container can hold large amounts of data, but typically it is used to hold small pieces of data because it is often implemented for every request through a handler in the global.asax.

The Context container (accessible from the Page object or using System.Web.HttpContext.Current) is provided to hold values that need to be passed between different HttpModules and HttpHandlers. It can also be used to hold information that is relevant for an entire request. For example, the IBuySpy portal stuffs some configuration information into this container during the Application_BeginRequest event handler in the global.asax. Note that this only applies during the current request; if you need something that will still be around for the next request, consider using ViewState.

Setting and getting data from the Context collection uses syntax identical to what you have already seen with other collection objects, like the Application, Session, and Cache. Two simple examples are shown here:

// Add item to Context
Context.Items["myKey"] = myValue;

// Read an item from the Context
Response.Write(Context["myKey"]);

ViewState

ViewState holds the state information for a single user, for as long as he is working with this ASPX page. The ViewState container can hold large amounts of data, but care must be taken to manage the size of ViewState since it adds to the download size of every request and response.

ViewState is one of the new containers in ASP.NET that you're probably already using, even if you don't know it. That's because all of the built-in Web controls use ViewState to persist their values between page postbacks. This is all done behind the scenes, so you don't need to worry about it most of the time. You should be aware of it, though, since it does impose a performance penalty on your application. How big this penalty is depends on how much ViewState you are carrying between postbacks—for most Web forms the amount of data is quite small.

The easiest way to determine the amount of ViewState being used by each control on a page is to turn on page tracing and examine how much ViewState each control is carrying. If a particular control doesn't need to have its data persisted between postbacks, turn off ViewState for that control by setting EnableViewState to false. You can also see the total size of the ViewState on a given ASP.NET page by viewing the HTML source of the page in a browser and examining the hidden form field, __VIEWSTATE. Note that the contents are Base64-encoded to prevent casual viewing and manipulation. ViewState can also be disabled for an entire page by adding EnableViewState="false" to the @Page directive.

A typical Web form won't need to manipulate ViewState directly. If you build custom Web controls, however, you will want to understand how ViewState works and implement it for your controls so that they work similarly to the Web controls that ship with ASP.NET. Reading and writing values to and from ViewState is done using the same syntax I've used for the other collection objects:

// Add item to ViewState
ViewState["myKey"] = myValue;

// Read an item from the Context
Response.Write(ViewState["myKey"]);

When building your own custom Web controls, you may also want them to take advantage of ViewState. This is simply done at the property level in your controls. Figure 10 shows how you might store the PersonName property of a simple custom control in ViewState, and use it in the control's Render method.

Figure 10 Storing a Property in ViewState (Close Figure 10)
namespace MSDN.StateManagement
{
    public class HelloPerson : System.Web.UI.Control
    {
        public string PersonName
        {
            get
            {
                string s = (string)ViewState["PersonName"];
                return ((s == null) ? "" : s);
            }
            set
            {
                ViewState["PersonName"] = value;
            }
        }
        protected override void Render(System.Web.UI.HtmlTextWriter
            writer)
        {
            writer.Write("Hello " + PersonName);
        }
    }
}

Web.config and Machine.config Files

The data in these files is available to all users of an application. Data stored in the Web.config file is available for the life of the application. The data is generally small and the object works well for holding strings for file locations and database connections. Larger pieces of data are better kept elsewhere.

In addition to the various collection objects available, ASP.NET introduces a set of XML configuration files that are used to manage many of the settings for your applications, and even for your whole server. Each ASP.NET application uses a Web.config file to set many of its properties, and each server has a Machine.config file located in its system folder that is used as a basis for all applications. These settings are used as defaults unless overridden. In addition to storing configuration data, these files can store data that your application (or many applications, in the case of the Machine.config) needs.

Configuration information is read whenever the application starts, and is then cached. Since it is cached, the application can read this data very quickly, so you should not be concerned that your application will have a bottleneck because it has to constantly refer to a text file for some integral information. In addition, changes to the Web.config result in an application restart for that application (or all applications on the machine with Machine.config). This ensures that changes to configuration information are always reflected immediately by the application.

Database connection information, default image paths, and paths to XML data files are some of the more common pieces of data that are stored in the Web.config file. The syntax for storing data in the Web.config file is as follows, although ideally you might want to use integrated SQL authentication:

<configuration>
    <!-- application specific settings  -->
    <appSettings>
        <add key="connectionString" value="server=myDBServer;
            uid=myUID;pwd=myPassword;database=myDB" />
    </appSettings>
    <system.web>
        <!-- all wsb settings go here -->
    </system.web>
</configuration>

To access these values within your ASP.NET pages, you use the ConfigurationSettings collection, which is in the System.Configuration namespace. The following simple example demonstrates how to extract the previous connection string into a local variable:

using System.Configuration;
•••
String strConnString =
  ConfigurationSettings.AppSettings["connectionString"];

Adding a reference to the System.Configuration namespace reduces the amount of code required to reference these values. Since changes to the Web.config or Machine.config result in an immediate application restart, these values are typically only modified by the server administrator, usually by hand. Thus, you should think of these files as being a good place to store read-only data, not data that you will need to modify within your application.

Conclusion

Effective state management can mean the difference between a frustrating user experience with the potential for data corruption and a smooth, fast page or transaction process. While state management in ASP 3.0 was somewhat unwieldy, ASP.NET brings it under control with the state objects discussed here. With their careful use, you'll be on your way to presenting the best Web experience possible to your customers.

For related articles see:
Cutting Edge: Using Session and Application Objects in ASP .NET
ASP.NET Caching

For background information see:
Caching Application Data
Application Cache Viewer
Ultimate Caching: Output and Fragment Options
ASP.NET Caching Features

Steven A. Smithis president of ASPAlliance.com, LLC, a popular resource for the ASP and ASP.NET developer community. He conducts ASP.NET training for ASPSmith.com, LTD. Steve is a coauthor of the ASP.NET Developer's Cookbook (SAMS, 2003). He can be reached at ssmith@aspalliance.com.


ADO.NET: Tackle Data Concurrency Exceptions Using the DataSet Object

ADO.NET

Tackle Data Concurrency Exceptions Using the DataSet Object

David Burgett
This article assumes you're familiar with ADO.NET and Visual Basic .NET
Level of Difficulty 1 2 3
SUMMARY

ADO.NET provides many techniques for improving the performance of data-intensive applications and for making them easier to build. The DataSet, the hallmark of the ADO.NET object model, serves as a miniature, disconnected facsimile of a data source. While using the DataSet improves performance by reducing expensive trips to the database server, it also introduces the possibility of multiple users attempting to change the same data simultaneously, thereby generating data concurrency exceptions. This article examines the common causes behind data concurrency exceptions and presents techniques for overcoming them.

Contents
DataSet Pros and Cons
The Sample Application
The First Data Concurrency Exception
The Solution
Using SQL Server Triggers
Conclusion

T
here are many benefits to upgrading your data access layer to ADO.NET, most of which involve using the intrinsic DataSet object. The DataSet object is basically a disconnected, in-memory replica of a database. DataSet objects contain one or more DataTables, each of which usually corresponds to a table in the underlying database. DataSets provide many benefits, but also present a few challenges. Specifically, you can run into problems related to data concurrency exceptions. I've created a simple Windows® Forms customer service application that illustrates the potential pitfalls of this particular problem. In this article, I'll walk you through the app and show you ways to overcome the data concurrency issues that arose.

The customer service application example shown here is built using Visual Basic® .NET and SQL Server™ 2000; however, since the Microsoft® .NET Framework is language-agnostic, any .NET Framework-compatible language could be used. Likewise, since the DataSet object abstracts the data source, the actual implementation of the data source is not important; data concurrency exceptions will be generated identically whether the underlying data source is SQL Server, local XML files, or derived from data retrieved through a service.

DataSet Pros and Cons

DataSets provide a number of benefits. For example, you gain the ability to enforce rules of integrity in memory rather than at the database level. DataSet objects can define and enforce relationships between tables and constraints upon columns, ensuring that business rules are applied without a round-trip to the database. With database abstraction, you can write a single set of code to access a DataSet object regardless of the origin of the data that fills the DataSet. The underlying data source may be SQL Server, Oracle, or even XML files. Regardless of the underlying data source, your code interacts with the DataSet in the same way. This allows you to change the underlying data source without changing your code.

The most important benefit of using DataSets, however, is improved performance. Since the DataSet is disconnected from the underlying database, your code will make fewer calls to the database, significantly boosting performance. You can add many new rows to multiple DataTables in a DataSet, checking each row for validity and referential integrity as you create it. A DataAdapter object, which connects the DataSet to the underlying database, can then update the underlying data with a single command. All of the new rows in each table are added with that command, assuring that all of the rows being added to the database are valid.

As with most performance optimizations, this one comes with a price. Since the DataSet object is disconnected from the underlying database, there is always a chance that the data is out of date. Since a DataSet doesn't hold live data, but rather a snapshot of live data at the time the DataSet was filled, problems related to data concurrency can occur. Data concurrency issues arise when multiple users have access to the same data and any single user can update the data without the other users' knowledge. It then becomes possible for a user to accidentally update data without knowing that the data they are changing is not the same data they see in their application. Fortunately, the DataSet object comes with built-in support for catching data concurrency issues as they arise so that your application can react accordingly.

The Sample Application

The customer service application I've created is used by a fictional company to create orders on behalf of their customers, as well as update the customers' personal information. There are many customer sales representatives (CSRs) who use the application from their own desktop. These CSRs take orders over the phone, gathering personal and payment information from the customer. The customer record is saved in the database to speed the processing of orders placed by returning customers. The CSR then creates an order and adds individual products to it, specifying the quantity and current price. Once all the information is gathered, the CSR presses the Place Order button, which inserts the customer and order records into the actual database.

CSRs also use the application to fulfill requests sent to the company through electronic or snail mail. These requests are divided evenly among the CSRs and forwarded to them each morning. CSRs fulfill the requests between phone calls. The system is designed to improve the speed at which requests are fulfilled; however, it comes at the cost of having all customers shared among CSRs. Each request made by a single customer, whether by phone or mail, will likely be processed by a different CSR, increasing the chance of data concurrency issues.

In order to improve performance, the application keeps a DataSet object filled with customers and orders in memory. Since many employees use the application simultaneously, there are always many disconnected snapshots of the live data, one on each employee's workstation. All customer maintenance, order entry, and order maintenance use this DataSet object, named dsAllData. The code used to build dsAllData is shown in Figure 1 and is part of a global module, so it's available to all forms in the application.

Figure 1 Filling the DataSet Object (Close Figure 1)
Const connString = "server=localhost;database=northwind;uid=sa;pwd="
    Public connCustSvc As SqlClient.SqlConnection
    Public daCustomer As SqlClient.SqlDataAdapter
    Public cbCustomer As SqlClient.SqlCommandBuilder
    Public daOrders As SqlClient.SqlDataAdapter
    Public cbOrders As SqlClient.SqlCommandBuilder
    Public daOrderDetail As SqlClient.SqlDataAdapter
    Public cbOrderDetail As SqlClient.SqlCommandBuilder
    Public dsAllData As DataSet
    Public Sub Main()
        connCustSvc = New SqlClient.SqlConnection(connString)
        daCustomer = New SqlClient.SqlDataAdapter("SELECT * FROM
                         Customer", connCustSvc)
        cbCustomer = New SqlClient.SqlCommandBuilder(daCustomer)
        daOrders = New SqlClient.SqlDataAdapter("SELECT * FROM Orders",
                       connCustSvc)
        cbOrders = New SqlClient.SqlCommandBuilder(daOrders)
        daOrderDetail = New SqlClient.SqlDataAdapter("SELECT * FROM
                            OrderDetail", connCustSvc)
        cbOrderDetail = New SqlClient.SqlCommandBuilder(daOrderDetail)
        dsAllData = New DataSet()
        daCustomer.MissingSchemaAction = MissingSchemaAction.AddWithKey
        daCustomer.Fill(dsAllData, "Customer")
        daOrders.MissingSchemaAction = MissingSchemaAction.AddWithKey
        daOrders.Fill(dsAllData, "Orders")
        dsAllData.Tables("Orders").Columns("Total").DefaultValue = 0
        daOrderDetail.MissingSchemaAction =
            MissingSchemaAction.AddWithKey
        daOrderDetail.Fill(dsAllData, "OrderDetail")
        Application.Run(New frmCustomerMaintenance())
    End Sub

The code that builds dsAllData creates an empty DataSet object, three DataAdapters, and three CommandBuilders. Each DataAdapter does a simple "SELECT *" from the appropriate table, while the CommandBuilder fills in the rest of the information necessary for the DataSet to have full insert, update, and delete capability. The Main routine uses the DataAdapter objects to fill dsAllData with data from all three tables and then begins the application with the Customer Maintenance form.

Figure 2 shows the Customer Maintenance screen, which has a DataGrid control bound to the Customers DataTable in dsAllData. This simple grid allows CSRs to edit any of the basic properties of a customer. Since the grid is bound to the Customers DataTable, any changes made to the values in the grid will automatically be stored in the DataTable. dsAllData will store these values until the CSR explicitly tells the form to update the underlying data source by clicking the Save Changes button.

Figure 2 Customer Maintenance Grid Allows Editing
Figure 2 Customer Maintenance Grid Allows Editing

For order entry, several new rows are created and added to dsAllData using the code in Figure 3. First an Order record is created, and then records are created in the OrderDetail DataTable for each item in the order. When all of the necessary rows have been added to dsAllData, a single call to the Update method of the appropriate DataAdapter will update the underlying data source with the new rows.

Figure 3 New Order with Detail Records (Close Figure 3)
Private Sub CreateOrder()
    Dim dr As DataRow
    dr = dsAllData.Tables("Orders").NewRow
    With dr
        .Item("DateOrdered") = Now
        .Item("CustomerID") = 1
        .Item("ShipToAddress") = "123 Main"
        .Item("ShipToCity") = "Kansas City"
        .Item("ShipToState") = "MO"
        .Item("ShipToZip") = "12345"
    End With
    dsAllData.Tables("Orders").Rows.Add(dr)
    AddOrderDetail(dr.Item("ID"), 1, 1, 9.99)
    AddOrderDetail(dr.Item("ID"), 2, 2, 4.99)
    daOrders.Update(dsAllData.Tables("Orders"))
    daOrderDetail.Update(dsAllData.Tables("OrderDetail"))
End Sub

Private Sub AddOrderDetail(ByVal OrderID As Integer, _
    ByVal ProductID As Integer, ByVal Quantity As Integer, _
    ByVal Price As Single)

    Dim dr As DataRow
    dr = dsAllData.Tables("OrderDetail").NewRow
    With dr
        .Item("OrderID") = OrderID
        .Item("ProductID") = ProductID
        .Item("Quantity") = Quantity
        .Item("Price") = Price
    End With
    dsAllData.Tables("OrderDetail").Rows.Add(dr)
End Sub

Since there are many CSRs using the application and updating customer information simultaneously, it is likely that at any given time a CSR is looking at out-of-date information. Planning for this, the architect of the application decided that the data cached in dsAllData should be refreshed every 30 minutes in order to ensure that CSRs are usually looking at correct information. The application is carefully architected to ensure that the data refresh is only performed during idle time so that it doesn't affect performance. DataSet refreshes are therefore slightly more than 30 minutes apart, depending on the activity of the CSR.

Figure 4 Application Data Model
Figure 4 Application Data Model

The data model for this application is very simple (see Figure 4). It is stored in SQL Server 2000 and consists of just three tables, one for customers, one for orders, and one for the details of each order. The appropriate primary keys and relationships have been defined to ensure referential integrity. Additionally, a trigger has been defined on the OrderDetail table to update the Total column of the Orders table. Each time an OrderDetail record is inserted, updated, or deleted, the trigger will fire, calculate the total sale value of the order, and update the appropriate row of the Orders table. The code for the trg_UpdateOrderTotal is shown in Figure 5.

Figure 5 Updating the Total Column (Close Figure 5)
CREATE TRIGGER trg_UpdateOrderTotal ON [dbo].[OrderDetail]
FOR INSERT, UPDATE, DELETE
AS
DECLARE @OrderID int
SELECT @OrderID=OrderID FROM Inserted
IF @OrderID IS NULL
    SELECT @OrderID=OrderID FROM Deleted
UPDATE Orders
SET Total=
    (
    SELECT Sum(Price*Quantity)
    FROM OrderDetail
    WHERE OrderID=@OrderID
    )
WHERE ID=@OrderID

The First Data Concurrency Exception

The customer service application has been running in production with no problems for a couple of months when suddenly the application goes down with an unhandled DBConcurrencyException. In this section, I'll break down the circumstances that led to this new exception.

Joe, the first CSR, opens the application. This forces the initial load of data into the dsAllData DataSet and begins the cycle of refreshing the data every 30 minutes or so. Joe has a stack of paperwork in his inbox containing change requests that customers have faxed, mailed, or e-mailed. He begins working through the change requests, but is frequently interrupted by phone call orders.

Meanwhile, Sally, the second CSR, arrives at the office and opens her own instance of the application. Sally's instance of the app does an initial load of data from SQL Server, including any updates Joe has made so far. Sally receives a phone call from a customer requesting a phone number change. The customer had e-mailed a change of address request previously, but didn't know their new phone number at that time and wants to update their record now. Sally uses the Customer Maintenance screen to update the customer phone number. When Sally changes the phone number in the DataGrid, the new number is stored in dsAllData. When Sally confirms the rest of the customer's information, she realizes that the original address change request has not yet been processed so she updates the information and clicks the Save Changes button, which sends the new data to the SQL Server database.

Joe is in the next cubicle working diligently on his e-mail requests when he comes across the original change request for the same customer. When he pulls up his customer maintenance screen, the application works as designed and pulls the information from the cached DataSet object. Joe's application has not automatically synchronized itself with the database since Sally updated the customer's address, so his Customer Maintenance screen still shows the old address. Joe corrects the information displayed in the DataGrid with the new information provided in the e-mail and clicks the Save Changes button. When he does, however, he receives an error message that reads "Concurrency violation: the UpdateCommand affected 0 records," and the application crashes. When Joe restarts the application, he sees that the address has been updated and believes his change must have been completed before the application crashed. The following is the line of code in question:

    Private Sub butSave_Click (ByVal sender As System.Object, _
                               ByVal e As System.EventArgs)
        daCustomer.Update(dsAllData.Tables("Customer"))
    End Sub

The actual exception that is created is of type DBConcurrencyException and is the result of specific functionality built into the DataAdapter object (see Figure 6). The DataAdapter is designed to pull data into disconnected objects like the DataSet, so it automatically double-checks data for changes before performing an update. If the underlying data has been changed, the DataAdapter will then raise the DBConcurrencyException instead of performing the requested update.

Figure 6 A DBConcurrency Exception
Figure 6 A DBConcurrency Exception

The implementation of this integrity check is fairly straightforward and leverages the ability of the DataTable object to keep multiple sets of data. When data is first loaded into a DataTable, all DataRows in the DataTable have their DataRowVersion property set to Original. When you modify a column value in a DataRow, a copy of the DataRow is created and marked as Current. The DataRow marked Original still exists in its unchanged form. All subsequent changes to the data will only affect the Current DataRow. When you execute the DataAdapter's Update method for a DataTable (or multiple DataTables in a DataSet), it iterates through all Current rows to determine the Update statements that should be applied to the underlying data source. In addition to the DataRowVersion property, DataRows have a RowState property which identifies the state of data in the row. Possible values include Unchanged, Added, Modified, Deleted, and Detached.

Having now determined which rows of the underlying data require updating, the dsCustomer DataAdapter builds the SQL statements needed to update the SQL Server database. Recall from Figure 1 that I used CommandBuilder objects with my DataSet to build the necessary INSERT, UPDATE, and DELETE statements. The UPDATE statement created by the CommandBuilder object uses the values stored in the copy of the DataRow with a DataRowVersion value of Original to identify and update the appropriate row in the database. That is, instead of simply identifying the correct row using the primary key values, the CommandBuilder creates a SQL statement that looks for a row which exactly matches all of the original values stored in the DataSet. The following code is an example of the UPDATE statement built to update the customer's phone number:

UPDATE Customer SET Phone = @p1
WHERE ((ID = @p2) AND ((FirstName IS NULL AND @p3 IS NULL)
      OR (FirstName = @p4))
AND ((LastName IS NULL AND @p5 IS NULL) OR (LastName = @p6))
AND ((Address IS NULL AND @p7 IS NULL) OR (Address = @p8))
AND ((Address2 IS NULL AND @p9 IS NULL) OR (Address2 = @p10))
AND ((City IS NULL AND @p11 IS NULL) OR (City = @p12))
AND ((State IS NULL AND @p13 IS NULL) OR (State = @p14))
AND ((Zip IS NULL AND @p15 IS NULL) OR (Zip = @p16))
AND ((Phone IS NULL AND @p17 IS NULL) OR (Phone = @p18)))
The UPDATE statement uses parameters rather than actual values, but you can see how the value of each and every column in the row is checked.

Having now identified the exact row and original values in the underlying database, the DataAdapter can safely update the row. However, if any of the column values in the database row have changed since the DataTable was filled, the UPDATE statement will fail because there will be no row in the database that matches the criteria in the WHERE clause. It is easy for the DataAdapter to determine whether the UPDATE was successful by simply checking the actual number of rows updated in the database. If no rows are updated, then the underlying data must have been altered or deleted and the data concurrency exception is raised. This explains the somewhat cryptic error message that Joe receives when he tries to update the customer's phone number; the exact error detected by the DataAdapter is not that the underlying data changed, but that no records were updated, indicating that the underlying data must have changed.

The Solution

There are a couple of ways to deal with the DBConcurrencyException. The first is to ensure that it never occurs. I can eliminate the SqlCommandBuilder objects used in Figure 1 and replace them with custom-built SqlCommand objects for the UpdateCommand properties of my DataAdapter objects. I would create the SQL used in the CommandText property with a WHERE clause that only filters based on the primary key instead of all available columns. This would eliminate all concurrency issues, assuming that the primary key is immutable.

However, this technique introduces several problems. First, there is considerably more code involved since I would also have to create SqlCommand objects for the InsertCommand and DeleteCommand properties of each of my DataAdapters. Additionally, these hard-coded SQL statements would introduce new errors should the underlying database schema be altered. Using the SqlCommandBuilder objects, the application determines the database schema at run time, accepting any changes which may have been introduced, and creates the SQL statements accordingly. This does not solve the concurrency problem, but rather avoids the issue altogether, enabling users to unknowingly overwrite someone else's changes.

A better solution is to catch the exception, as shown in Figure 7. The Try...Catch block catches the DBConcurrencyException and displays a message box identifying the exception to the user and offering them an option, as shown in Figure 8. At this point, I have identified that a concurrency error has occurred and I have two choices: I can retrieve the new underlying data and display it to the user, forcing them to make their changes yet again, or I can simply overwrite the new underlying data with the changes that have been specified by the user. These are the options that are displayed in the message box shown in Figure 8.

Figure 7 Catching the Concurrency Exception (Close Figure 7)
Try
  daCustomer.Update(dsAllData.Tables("Customer"))
Catch dbcEx As Data.DBConcurrencyException
  Dim dResult As DialogResult
  dResult = MessageBox.Show(messageString, _
    "Data Concurrency Exception Occurred", _
    MessageBoxButtons.YesNoCancel, MessageBoxIcon.Error, _
    MessageBoxDefaultButton.Button1, _
    MessageBoxOptions.DefaultDesktopOnly)
  If dResult = DialogResult.Yes Then
    'Two options: Fill the whole table, or just refresh the row
    'daCustomer.Fill(dsAllData.Tables("Customer"))
    UpdateRow("Customer", dbcEx.Row.Item("ID"))
  ElseIf dResult = DialogResult.No Then
    'Save a copy of the new row
    Dim drCopy As DataRow, drCurrent As DataRow
    drCopy = dsAllData.Tables("Customer").NewRow()
    Dim dc As DataColumn
    drCurrent =  _
      dsAllData.Tables("Customer").Rows.Find(dbcEx.Row.Item("ID"))
    For Each dc In drCurrent.Table.Columns
      If dc.ReadOnly = False Then _
        drCopy.Item(dc.ColumnName) = drCurrent.Item(dc.ColumnName)
    Next

    'Get the current values from the database
    UpdateRow("Customer", dbcEx.Row.Item("ID"))

    'Now restore the values entered by the user and save again
    For Each dc In drCurrent.Table.Columns
      If dc.ReadOnly = False Then _
        drCurrent.Item(dc.ColumnName) = drCopy.Item(dc.ColumnName)
    Next

    daCustomer.Update(dsAllData.Tables("Customer"))
  End If
End Try
Figure 8 Handling the Data Concurrency Exception
Figure 8 Handling the Data Concurrency Exception

If the user decides to view the new underlying changes and discard their current changes, you simply need to refresh the data stored in the Customer DataTable. Since the DataGrid is bound to this DataTable, the grid will automatically display the new data. To refresh the data, you have two options. The first is to simply fill the entire DataTable again using the Fill method of the daCustomer DataAdapter. While this technique will work, it is overkill since you are only interested in refreshing the data in a single DataRow. I have created a routine named UpdateRow which only pulls data for the single DataRow in question (see Figure 9). One caveat of using the UpdateRow routine is that I have defined in the parameter collection that the DataRow to be found will have a single, integer key column. If the table has a key of a different data type or has a composite key, you will need to overload UpdateRow to handle the specific key requirements. After the DataRow and/or DataTable have been refreshed with the current data, the DataGrid displays a small Error icon next to the row that caused the concurrency exception (see Figure 10).

Figure 9 UpdateRow Routine Updates Cached DataRow (Close Figure 9)
Private Sub UpdateRow(ByVal TableName As String, ByVal ID As Integer)
  'Get a reference to the specified row
  Dim dr As DataRow = dsAllData.Tables(TableName).Rows.Find(ID)

   'Create a Command update to pull the new underlying data
  Dim cmd As New SqlClient.SqlCommand("SELECT * FROM " & TableName & _
   " WHERE ID=" & ID.ToString(), connCustSvc)

  'Open the connection and create the DataReader
  connCustSvc.Open()
  Dim rdr As SqlClient.SqlDataReader = cmd.ExecuteReader()
  rdr.Read()

  'Copy the new data from the database to the DataRow
  Dim dc As DataColumn
  For Each dc In dr.Table.Columns
    If dc.ReadOnly = False Then _
     dr.Item(dc.ColumnName) = rdr.Item(dc.ColumnName)
  Next

  'Accept changes in the DataRow
  dr.AcceptChanges()
  connCustSvc.Close()
End Sub
Figure 10 DataGrid Displays Error Icon
Figure 10 DataGrid Displays Error Icon

The other option is for the user to disregard the changes made to the underlying database and force their changes to take effect. There are a couple of ways this can be accomplished. The first would be to execute a direct SQL statement against the SQL Server database to synchronize it with the data in the DataTable. While this method performs well, it can be problematic in that it would cause you to rewrite the SQL statement with every database change. The SQL you write for this technique may also need to be hard-coded to the specific database version being used, losing the abstraction afforded to you by the DataAdapter and DataSet objects.

A better option uses the UpdateRow routine I wrote previously and allows the DataAdapter to handle the update. The code in Figure 9 first creates a copy of the DataRow with the new changes and then calls the UpdateRow routine to get the new data from the database. It is necessary to call the UpdateRow procedure so that you do not receive another concurrency exception when you try to perform the update again. The code then iterates through all the columns on the DataRow, replacing the newly retrieved values with those supplied by the user. Finally, the DataAdapter is used to update the underlying database with the user's changes.

There are a couple of potential problems with both of these code solutions. First, by default, the DataAdapter's Update method will fail on the first concurrency exception and will not process additional DataRows. This could lead to partial updates of data or several concurrency exceptions, each handled individually by the user. Another potential exception could arise in the code shown in Figure 7, in the branch that forces the user's changes into the underlying database. There is a slim chance that another user may change the underlying database between the call to the UpdateRow routine and executing the Update method on daCustomer. This would generate an unhandled DBConcurrencyException. One possible workaround is to place the Update method call in a Try...Catch block, but your second Catch code would be similar to the first and could potentially generate its own concurrency exception, requiring more Try...Catch blocks, ad infinitum. A more elegant solution would break this code out into a separate routine which could be called recursively in the case of multiple concurrency exceptions.

Using SQL Server Triggers

The developers of the customer service application for my fictional company have now fixed the code to handle the DBConcurrencyException that Joe received when updating a customer, and everything is working fine again—that is, until Sally tries to enter an order later that afternoon.

Sally receives a phone call from a customer wanting to place an order. The customer is already in the database, so Sally takes the basic information for the order, including the shipping address. She then pulls up the OrderDetails screen and adds two items to the order. Each OrderDetail record includes an OrderID, a ProductID, a Quantity, and a Price. When the order is correct, Sally clicks the Place Order button to insert the order into the database. I have simplified the code by adding hard-coded values for the OrderDetail records in the code shown in Figure 3.

The order is successfully placed and inserted into the database, but Sally receives a notification that one of the items is currently out of stock and the order cannot be shipped for two weeks. The customer realizes that he will be out of town when the package is available for delivery and asks if he can change the shipping address for the order. Sally has changed information about an order before by using the Order Maintenance screen, so she says that she'll be happy to help. She pulls up the appropriate screen and changes the shipping address for the customer. However, when she clicks the Save Changes button, she sees an error dialog similar to the one that Joe saw earlier, and the app crashes.

The key to understanding why this DBConcurrencyException is raised lies in the underlying database. Recall from Figure 5 that I placed a trigger on the OrderDetail table that is fired for all INSERTS, UPDATES, and DELETES. The trg_UpdateOrderTotal updates the total cost of the order by calculating the sum of the Quantity times the Price for each OrderDetail row tied to the Order. When Sally created the order with two items, the application first inserted a new row into Orders and then inserted two new rows into OrderDetail. After each of the OrderDetail records is created, the trigger is fired, which updates the Total column of the Order record. However, this value is generated only in the database and is not automatically transferred to the application. In fact, the code in Figure 3 does not specify a Total because a default value was already specified in Figure 1. The default value of 0 is then passed into SQL Server with the new Orders record, which is correct at the time the order is created. The database then updates the total column, but the DataTable in the application still has 0 as the order total. When Sally tries to update the Order record, the DataAdapter recognizes that the Total column value has changed and raises the DBConcurrencyException.

The trick here is that in the past when Sally had updated Order records, by chance, she didn't update a new order that had been created since the last automatic data refresh had taken place. Within 30 minutes of creating each order, the dsAllData DataSet object is refreshed, which will update the Orders DataTable with the correct totals. Any order updates made after the automatic data refresh (or an application restart) will work as planned.

The solution to this issue is similar to the previous problem: refresh the data seen by the user. I can, however, be a bit more proactive here. Instead of waiting for a concurrency exception to occur, I can automatically refresh each new Orders DataRow after OrderDetail records are created. I can modify the code in Figure 3 to include a call to my UpdateRow routine for the specified row in Orders just after calling the Update method of daOrderDetail. SQL Server will normally complete this trigger almost instantaneously, but you can't depend on this, so a prudent developer might add a time delay so the trigger has enough time to complete.

Conclusion

The disconnected nature of the new ADO.NET DataSet object offers significant performance benefits at the cost of possibly introducing new types of data concurrency errors into your application. Understanding how and why these errors occur allow you to catch and handle such errors elegantly, but also adds an additional level of support to your application. This gives your users more options for maintaining the integrity of their underlying database. Handling concurrency issues is sure to become an increasingly larger challenge as application development moves farther into the world of Internet and intranet applications, with hundreds or thousands of simultaneous users. ADO.NET provides you with all the necessary tools to overcome this challenge.

For related articles see:
Data Points: DataRelations in ADO.NET
Advanced Basics: Viewing the Values of a DataSet in a Debug Window
Dataset Updates in Visual Studio .NET

For background information see:
.NET e-Business Architecture by David Burgett, Matthew Baute, John Pickett, and Eric Brown (SAMS, 2001)
Data Points: Establishing Relationships Between Rowsets with ADO.NET

David Burgettis a Senior Technical Architect with G. A. Sullivan in Kansas City, and the co-owner/developer of ReprintOrders.com. He has contributed to three books on Visual Studio, most recently .NET eBusiness Architecture (Sams, 2001). David frequently speaks at technical events. You can reach him at david@burgett.com.


Advanced T-SQL: Automate the Generation of Stored Procedures for Your Database

Advanced T-SQL

Automate the Generation of Stored Procedures for Your Database

Peter W. DeBetta and J. Byer Hill
This article assumes you're familiar with T-SQL
Level of Difficulty 1 2 3
Code download available at: StoredProcedures.exe (108KB)
SUMMARY

Design-time automation makes coding faster and ensures that all the procedures generated use the same naming conventions and structure. In an effort to improve their coding efficiency in a large SQL project, the authors wrote a set of design-time stored procedures that generate run-time stored procedures, and have used them in project after project ever since. Recently, the authors updated their procedures to make use of SQL Server 2000 features, including user-defined functions. This article covers the creation and execution of these dynamic T-SQL scripts to automate the coding of common database stored procedures.

Contents
A Simple Example
SQL Server System Tables and Views
Parsing a Table's Columns
Finding Primary Key Columns
Column Default Values
Dynamically Execute T-SQL
Creating Stored Procedures
Conclusion

S
ome time ago, we began development on a rather large n-tier client/server project. During the initial planning, we decided to require a set methodology for accessing the numerous tables in the database. Four base stored procedures would be used to perform selects, inserts, updates and record deletions on each table. Although the required set of stored procedures would be similar in design, each table's unique column structure would mandate that significant details of each individual procedure would vary, thus making the writing of all the required procedures very tedious indeed. It became quite clear that the stored procedure writing process had to be automated.

What started out as an experiment turned into a core set of design-time stored procedures that write the base set of run-time stored procedures for all database tables in any given project. By building and using these design-time procedures, we not only saved ourselves hours of tedium, but saved our client some money as well, and came away with some nifty code that we still use today.

These design-time procedures have been updated to take advantage of some of the features of SQL Server™ 2000, specifically, user-defined functions (UDFs). So now the code is more modular, and we have additional functions available to us for other tasks.

Another bonus that design-time automation offers is the guarantee that the hundreds of procedures generated would be consistently structured and have a standard naming convention. In our case, all run-time procedure names that are generated are formatted as such: prApp_TableName_Task, where Task can be either Select, Insert, Update, or Delete. The procedures for the Customers and Orders tables would look something like this:

prApp_Customers_Delete
prApp_Customers_Insert
prApp_Customers_Select
prApp_Customers_Update
prApp_Orders_Delete
prApp_Orders_Insert
prApp_Orders_Select
prApp_Orders_Update

As you can see, this convention adds a great deal of organization to our database, making any given procedure easy to locate and making each procedure name self-describing. Developers will be able to quickly find and create code. And to top it off, future team members on the project will find the code and procedures easy to follow. Of course, if you already have a different naming convention in place, simply change a few lines of code and your conventions can be used instead.

These four design-time procedures are not set in stone, but are meant to act as a template for use in other projects. They are installed in the project database, and if needed, they are modified to suit the needs of the specific application. For example, in several of our applications, we added code to maintain an audit trail record in a separate database every time a record was modified.

A Simple Example

Before we begin, let's take a look at a simple example using the Order_Details table (whose name has been changed to replace a space with an underscore character) from the Northwind database. (Although spaces and other characters are allowed in object names, we recommend that you use regular identifiers for object names to prevent issues when using these automated stored procedures. See "Using Identifiers" in SQL Server Books Online for more information.)

The first task is to run the design-time procedure in order to create the run-time procedure that updates data in the Order_Details table:

EXEC pr__SYS_MakeUpdateRecordProc 'Order_Details'

Running the design-time procedure will produce the T-SQL script shown in Figure 1 as output. When this T-SQL script is run, it creates a new Update stored procedure for the Order_Details table. All columns are accounted for as parameters in the new stored procedure code, but notice how the primary key columns (OrderID and ProductID) show up in the Where clause, while the other non-key fields are part of the Update statement's Set clause. Our design-time procedure examined the Order_Details metadata stored in the SQL Server system tables and used this info to create the appropriate T-SQL script as output, which will then create the final run-time procedure when executed.

Figure 1 Generated T-SQL Script (Close Figure 1)
IF EXISTS(SELECT * FROM sysobjects WHERE name =
    'prApp_Order_Details_Update')
    DROP PROC prApp_Order_Details_Update
GO
-- Update a single record in Order_Details

CREATE PROC prApp_Order_Details_Update
    @OrderID int,
    @ProductID int,
    @UnitPrice money,
    @Quantity smallint,
    @Discount real
AS

UPDATE    Order_Details
SET       UnitPrice = @UnitPrice,
          Quantity = @Quantity,
          Discount = @Discount
WHERE     OrderID = @OrderID
AND       ProductID = @ProductID

GO

This execution only produced output and not the new run-time procedure. With a simple modification, however, the design-time procedure can in fact execute the T-SQL script it generated as a final step. To do this, we simply run our design-time procedure, pass the value 1 as a bit flag for the second optional parameter, and execute it again:

EXEC pr__SYS_MakeUpdateRecordProc 'Order_Details', 1
This will not only show the output as before, but also execute the output, thus creating the run-time procedure.

Now let's take a look at the design-time procedure code that created this new application-specific run-time stored procedure.

SQL Server System Tables and Views

In order to create these design-time stored procedures, we must know how to extract the table's column definition information from the SQL Server system tables and Information Schema views. First, we need to find the columns themselves, and then find out which are the primary key columns, what datatype each column supports, and whether the column allows nulls.

Figure 2 View System Tables
Figure 2 View System Tables

It might be helpful to know that SQL Server Enterprise Manager allows you to view the system tables by changing a property of the server registration, as shown in the dialog in Figure 2. If you right-click the server in Enterprise Manager and then select "Edit SQL Server Registration properties," you will be presented with a dialog. At the bottom of the dialog, you will see a checkbox that's labeled "Show system databases and system objects." Checking that option turns it on to view system objects, or you can choose to turn it off and keep your table list view simpler and easier to read.

Parsing a Table's Columns

The syscolumns table provides much of the necessary metadata information, such as the column name, ID, length, and nullability. It is also used in conjunction with the sysindexes table to determine the primary key field(s) of the table. You will also see that the INFORMATION_SCHEMA.COLUMNS view will be used to retrieve the default values of our columns.

Since all procedures are looking for the same metadata information, it would be nice to encapsulate this in a separate piece of code for modularity and maintainability. As we stated before, earlier versions of SQL Server did not have UDFs, which made modularity like this a difficult task. But, with SQL Server 2000 user-defined functions, we decided to take the code a step further and modularize the common features of our four design-time stored procedures. Five new UDFs were created to deal with the system tables and information schema views, encapsulating all the metadata retrieval.

In order to properly create the new run-time stored procedures, we need to know the following metadata column information for the table in question:

  • The name of the column
  • The ID of the column
  • The datatype of the column
  • The maximum length of the column (applicable to character and binary data)
  • The precision of the column—or how many digit places its values have (for decimal and numeric data)
  • The scale of the column—or how many places exist after the decimal point (for decimal and numeric data)
  • Whether the column is nullable
  • Whether the column is part of the primary key
  • Whether the column is an Identity column
  • The default value of the column

For the most part, this information comes from the syscolumns table, with a couple of exceptions. The default value is actually retrieved from the INFORMATION_SCHEMA.COLUMNS view. The datatype name is extracted from the systypes table, and a more complex combination of syscolumns, sysindexes, and sysindexkeys is used to determine if a column is part of the primary key. It was so complex, in fact, that we encapsulated this functionality into its own UDF.

Let's take a look at the main function shown in Figure 3, which reveals more of this metadata information. This UDF is not overtly complex. As you can see, most of the metadata information is returned—except for some simple column renaming—without any modification, including column name, colid, length, precision, scale, IsNullable, and type name. As for the rest, a little more work needs to be done for this information. For the primary key metadata, we have created another UDF that determines whether a column is part of the primary key of a table. We will examine both of these additional UDFs shortly.

Figure 3 UDF (Close Figure 3)
CREATE FUNCTION dbo.fnTableColumnInfo(@sTableName varchar(128))
RETURNS TABLE
AS

RETURN
SELECT    c.name AS sColumnName,
    c.colid AS nColumnID,
    dbo.fnIsColumnPrimaryKey(@sTableName, c.name) AS bPrimaryKeyColumn,
    CASE
     WHEN t.name IN ('char', 'varchar', 'binary', 'varbinary', 'nchar',
                     'nvarchar') THEN 1
     WHEN t.name IN ('decimal', 'numeric') THEN 2
     ELSE 0
    END AS nAlternateType,
    c.length AS nColumnLength,
    c.prec AS nColumnPrecision,
    c.scale AS nColumnScale,
    c.IsNullable,
    SIGN(c.status & 128) AS IsIdentity,
    t.name as sTypeName,
    dbo.fnColumnDefault(@sTableName, c.name) AS sDefaultValue
FROM    syscolumns c
    INNER JOIN systypes t ON c.xtype = t.xtype and
         c.usertype = t.usertype
WHERE    c.id = OBJECT_ID(@sTableName)

Let's take a look at the alternate type and the identity status. The eighth bit (128) of the status field of syscolumns indicates whether or not the column is an identity. (This is especially important to know when creating Insert and Update scripts.) Our simple formula consists of performing a logical AND (&) with this value, and then taking that result and wrapping it in the Sign function. If the bit is set, meaning the column is an identity column, c.status & 128 will return a value of 128. Otherwise, a value of 0 will be returned. The Sign function returns a value of 1 for positive values, a negative one for negative values, and a zero for 0 values. So, if the column being evaluated is an identity column, a value of 1 will be returned, otherwise, 0 is returned.

The alternate type is used to indicate whether the datatype requires additional information (length or precision and scale) when being defined. We are categorizing character and binary datatypes as an alternate type of one, decimals and numerics as two, and all other datatypes as zero. This value is used in the stored procedures to determine if length or precision and scale values need to be added to the parameter definitions.

Finding Primary Key Columns

As you saw, finding column information is not too difficult. Finding out if a field is part of a primary key, however, requires a little more effort. There is a list of fields that can be retrieved, but these fields are found in a combination of the syscolumns, sysindexes, and sysindexkeys tables and must be compared to our requested column (passed in to the UDF in the @sColumnName parameter). Thus, the task of finding the primary key is easier to achieve in a separate user-defined function because we can encapsulate this work into a single function call.

Let's walk through this function and see what's really going on:

CREATE FUNCTION dbo.fnIsColumnPrimaryKey
    (@sTableName varchar(128), @sColumnName varchar(128))
RETURNS bit
AS
BEGIN
    DECLARE    @nTableID int,
               @nIndexID int,
               @i int

    SET @nTableID = OBJECT_ID(@sTableName)
The function takes two parameters, the table name and the column name, and returns a bit indicating if the specified column is part of a primary key for that table. We then need to declare some variables that will be used in the procedure and assign some initial values. Now comes the fun part: finding the primary key information. We begin by retrieving the Index ID for the primary key index of the table, as shown in the following code:
SELECT @nIndexID = indid
FROM      sysindexes
WHERE  id = @nTableID
 AND      indid BETWEEN 1 And 254
 AND      (status & 2048) = 2048
ORDER BY indid

IF (@nIndexID  Is Null)
    RETURN 0

We now assign the Index ID of the table's primary key index to the variable @nIndexID. The twelfth bit (2048) of the status column indicates if the index is a primary key index. If there is no primary key index, no records will be returned, leaving @nIndexID with a null value. So, if @nIndexID contains a null value, we leave the function, returning a 0 value. In other words, since there is no primary key index, the column cannot be part of a primary key. Now we need to check the requested column (@sColumnName) against the list of columns in the primary key index.

    IF @ColumnName IN
        (SELECT sc.[name]
          FROM       sysindexkeys sik
          INNER JOIN syscolumns sc ON sik.id = sc.id AND sik.colid =
                     sc.colid
          WHERE      sik.id = @nTableID
          AND        sik.indid = @nIndexID        )
     BEGIN
        RETURN 1
     END

    RETURN 0
END

Using the IndexID we retrieved earlier, we select the name of the column(s) from the join of syscolumns and sysindexkeys table. These tables are joined using the column ID and the object ID. The Where clause sets the criteria so that we only select columns from indexes related to our requested table (sik.id = @nTableID), and only for the primary key index (sik.indid = @nIndexID). If @sColumnName is in this retrieved list of columns, we return a value of 1; otherwise, we return a value of 0, which indicates that a match was not found.

Column Default Values

When a record is inserted into a table, if no value is supplied for a particular column and that column has a default value, the default is used to populate the column's data. Since our newly produced table-insert procedure has a parameter for all possible columns that can be inserted, and since a variable must contain a value, even if it is a null value, the table's defaults won't be used. Essentially, by explicitly supplying every column with a value (even null), we override the defaults of the columns. In order to counteract this characteristic of our created procedures, we need to be able to supply the defaults when inserting the data. We will see how the default values are used in our automated procedures a bit later in this article, but for now, let's examine how we get those default values in the first place.

The UDF that we will use simply references the built-in INFORMATION_SCHEMA.COLUMNS view, which supplies the default value of a column. It is easier to use this view to retrieve the default value than to use the sysconstraints system table. The next UDF will additionally simplify this process by wrapping the logic of finding the default into a simple function call:

CREATE FUNCTION dbo.fnColumnDefault(@sTableName varchar(128),
                                    @sColumnName varchar(128))
RETURNS varchar(4000)
AS
BEGIN
    DECLARE @sDefaultValue varchar(4000)

    SELECT  @sDefaultValue = dbo.fnCleanDefaultValue(COLUMN_DEFAULT)
      FROM      INFORMATION_SCHEMA.COLUMNS
      WHERE     TABLE_NAME = @sTableName
      AND       COLUMN_NAME = @sColumnName

    RETURN  @sDefaultValue

END

Column default values are stored wrapped in a pair of parentheses, which our implementation does not need. So as you can see, we pass the field COLUMN_DEFAULT to another function, fnCleanDefaultValue, which strips the parentheses off, and then returns the actual default.

For example, if a column named nQty has a default of 1, the COLUMN_DEFAULT value would actually contain (1). For a default of "Enter Text Here", we would get ("Enter Text Here"). Here is the source for that UDF:

CREATE FUNCTION dbo.fnCleanDefaultValue(@sDefaultValue varchar(4000))
RETURNS varchar(4000)
AS
BEGIN
    RETURN SubString(@sDefaultValue, 2, DataLength(@sDefaultValue)-2)
END
Now we have all the metadata information that we will need to actually create our automated procedures.

Dynamically Execute T-SQL

Dynamic T-SQL execution is the other essential feature of our stored procedures as it allows you to write a generic T-SQL script that in turn writes a T-SQL script. It is the T-SQL EXECUTE statement that allows the generic T-SQL script to actually execute its specific output and create the run time stored procedures to be used by the application.

EXECUTE, or EXEC, actually has two abilities: it can execute existing stored procedures and dynamically execute a SQL statement passed in as a string. It's the latter ability that we will be using in conjunction with the column metadata we extracted earlier to automatically create the stored procedures. A simplified view of the process would be filling a large varchar variable with the stored procedure code (using the metadata) required to create the stored procedure and then dynamically executing the contents of the varchar variable once complete, creating the new stored procedure.

Let's start by examining a simple example of dynamic T-SQL:

CREATE PROC prGetAuthor
    @au_id char(11)
AS
DECLARE @sExec varchar(8000)
SET @sExec = 'SELECT * FROM authors WHERE au_id = ''' + @au_id + ''''

EXEC (@sExec)
In this example, we pass in an author's ID and concatenate it with a SELECT statement that retrieves an author from the author's table.

Let's say we call the procedure like this:

EXEC prGetAuthor '123-45-6789'
The prGetAuthor procedure would create a SQL statement as shown in this line:
SELECT * FROM authors WHERE au_id = '123-45-6789'
This statement would then be executed in the EXEC statement and would return the author with the ID of 123-45-6789. As you are about to see shortly, our design-time procedures take this feature to a much higher level.

You probably already know this, but we should note that this is not a recommended use of dynamic T-SQL. Anytime dynamic T-SQL code is accessible to the outside world, the possibility of a SQL injection attack exists. We have made it a practice to use dynamic T-SQL for administrative and task purposes only, and never expose this functionality in any procedures that are accessible by anyone other than the system or an administrator.

Creating Stored Procedures

The first step in creating these design-time procedures is fairly standard. The procedure is defined, variables are declared, and some of them are initialized. A quick glance at this code reveals no surprises, but rather sets up the rest of the procedure. We do create two special string variables, one of which holds a tab character while the other contains a carriage return and line feed. These could also be set up as UDFs but we decided not to so that we could leave it as an exercise for the reader. These are used to help with the formatting of the code output. Let's take a look at the beginning of this procedure, which is shown in Figure 4.

Figure 4 Set up the Variables (Close Figure 4)
CREATE PROC pr__SYS_MakeUpdateRecordProc
    @sTableName varchar(128),
    @bExecute bit = 0
AS

IF dbo.fnTableHasPrimaryKey(@sTableName) = 0
 BEGIN
    RAISERROR ('Procedure cannot be created on a table with no primary
               key.', 10, 1)
    RETURN
 END

DECLARE    @sProcText varchar(8000),
    @sKeyFields varchar(2000),
    @sSetClause varchar(2000),
    @sWhereClause varchar(2000),
    @sColumnName varchar(128),
    @nColumnID smallint,
    @bPrimaryKeyColumn bit,
    @nAlternateType int,
    @nColumnLength int,
    @nColumnPrecision int,
    @nColumnScale int,
    @IsNullable bit,
    @IsIdentity int,
    @sTypeName varchar(128),
    @sDefaultValue varchar(4000),
    @sCRLF char(2),
    @sTAB char(1)

SET    @sTAB = char(9)
SET    @sCRLF = char(13) + char(10)

SET    @sProcText = ''
SET    @sKeyFields = ''
SET    @sSetClause = ''
SET    @sWhereClause = ''

Again, no great T-SQL revelations will be found here. Before we do anything else, we check to see if our table has a primary key value. This prevents our code from creating potentially damaging run-time procedures. Then we simply set up our variables and some default values. The next part of the procedure sets up the DROP statement for the new proc, just in case our new procedure already exists, creates some comments, and creates the actual procedure definition (the first few lines of Figure 1). You can modify this code to create a new run-time stored procedure only if it does not yet exist (and to do nothing if it already does). This new logic could also be controlled by the addition of a third optional parameter, @bIfExistsDoNothing. We will leave this as a simple exercise for the reader.

Our next code snippet begins the process of creating the dynamic T-SQL. Code is added to drop an existing procedure and for the new procedure definition (see Figure 5). Notice how we use the second (optional) parameter, @bExecute, to determine if we are going to actually run the code. In our automated procedure's definition, this parameter is optional, and defaults to a value of 0, meaning we would not actually execute the code.

Figure 5 Set Some Values (Close Figure 5)
SET     @sProcText = @sProcText + 'IF EXISTS(SELECT * FROM sysobjects
        WHERE name = ''prApp_' + @sTableName + '_Update'')' + @sCRLF
SET     @sProcText = @sProcText + @sTAB + 'DROP PROC prApp_' +
        @sTableName + '_Update' + @sCRLF
IF @bExecute = 0
    SET     @sProcText = @sProcText + 'GO' + @sCRLF

SET     @sProcText = @sProcText + @sCRLF

PRINT @sProcText

IF @bExecute = 1
    EXEC (@sProcText)

SET     @sProcText = ''
SET     @sProcText = @sProcText + '—' + @sCRLF
SET     @sProcText = @sProcText + '— Update a single record in ' +
        @sTableName + @sCRLF
SET     @sProcText = @sProcText + '—' + @sCRLF
SET     @sProcText = @sProcText + 'CREATE PROC prApp_' + @sTableName +
        '_Update' + @sCRLF

Next comes an interesting technique. We use the fnTableColumnInfo UDF as the source for a cursor, knowing that fnTableColumnInfo is a table-valued function. Instead of having to write the more complex T-SQL that this function uses in each of the four automated procedures, we now only have to reference the UDF in the cursor's declaration. After declaring the cursor, we then open it and fetch the first record into a set of variables that hold the metadata information, which we will use in order to create our new procedure (see Figure 6).

Figure 6 Declare Cursor and Read (Close Figure 6)
DECLARE crKeyFields cursor for
    SELECT    *
    FROM     dbo.fnTableColumnInfo(@sTableName)
    ORDER BY 2

OPEN crKeyFields

FETCH     NEXT
FROM      crKeyFields
INTO      @sColumnName, @nColumnID, @bPrimaryKeyColumn, @nAlternateType,
          @nColumnLength, @nColumnPrecision, @nColumnScale, @IsNullable,
          @IsIdentity, @sTypeName, @sDefaultValue

WHILE (@@FETCH_STATUS = 0)
 BEGIN

Of course, we will establish a loop, using the WHILE statement, that will continue as long as we have a valid row to retrieve (@@FETCH_STATUS = 0). Now we are ready to parse the column info and create the key segments for our new stored proc.

In the next code sample, we begin to loop through the cursor and use the column metadata to create the code. You will notice three key variables being modified: @sKeyFields, @sSetClause, and @sWhereClause. The first of these is used to create the parameter list for the stored procedure (inside the CREATE PRDC block of Figure 1). The second is used for the SET clause of the UPDATE statement in Figure 1. The last variable is used for the WHERE clause near the end of Figure 1. Now we should examine the first part of this code (see Figure 7).

Figure 7 Create the Parameter List (Close Figure 7)
IF (@sKeyFields <> '')
    SET @sKeyFields = @sKeyFields + ',' + @sCRLF

SET @sKeyFields = @sKeyFields + @sTAB + '@' + @sColumnName + '
                  ' + @sTypeName

IF (@nAlternateType = 2) —decimal, numeric
    SET @sKeyFields =  @sKeyFields + '(' + CAST(@nColumnPrecision AS
                       varchar(3)) + ', ' + CAST(@nColumnScale AS
                       varchar(3)) + ')'
ELSE IF (@nAlternateType = 1) —character and binary
    SET @sKeyFields =  @sKeyFields + '(' + CAST(@nColumnLength AS
                       varchar(4)) +  ')'

IF (@IsIdentity = 0)
 BEGIN
    IF (@IsNullable = 1) OR (@sTypeName = 'timestamp')
        SET @sKeyFields = @sKeyFields + ' = NULL'
 END

Figure 7 contains the code that creates the parameter list for our new stored procedure. The first IF statement checks to see if we have already appended data into our variable. If we have, we add a comma and a carriage return/line feed, so that we end each parameter in the parameter list correctly. If we did not do this check, and instead just appended the comma at the end of this segment, we would end up with one too many commas. By appending the comma before adding the next column, we prevent this problem.

Next, we append a tab character, an at symbol (@), the name of the column, a space, and the type name of the column—a simple concatenation of characters and metadata information. Then we have to see if we need additional information about the datatype. We check to see if the type requires information on precision, scale, or length. If any is required, we additionally append the value(s) wrapped in parentheses (as required by T-SQL syntax).

Finally, if the column is not an identity column, and the column accepts null values or is a timestamp (which is not allowed to be updated because it is updated directly), then we add "= NULL" to the parameter definition. For example, the columns from the discounts table in the pubs database would look like this:

discounttype varchar(40),
stor_id char(4) = NULL,
lowqty smallint = NULL,
highqty smallint = NULL,
discount decimal(4, 2)

Note that the discounts table has no primary key and would not allow proper automated code generation. These automated procedures rely on a primary key to determine how data will be updated. The automated procedure code could be modified to use all columns in the new procedure's WHERE clause if no primary key existed, or to look for a unique key and use its columns for the WHERE clause. In other words, tables should have primary keys if at all possible—it's a basic principle of database design.

Next, take a look at the code that creates the SET statement clause for our new procedure's UPDATE statement (see Figure 8). Notice how we only process columns that are not part of the primary key. Again, if you want to be able to update all columns, including those in the primary key, you could simply remove the IF statement. (Toggling this If logic on and off can be an added feature controlled by yet another optional parameter.) Like the last segment you saw, we append a comma to our variable, as needed. In this case, however, if the variable has no data (meaning we haven't added any columns yet), we put the SET clause in our variable to start it off.

Figure 8 Create the SET Clause (Close Figure 8)
IF (@bPrimaryKeyColumn = 0)
 BEGIN
    IF (@sSetClause = '')
        SET @sSetClause = 'SET'
    ELSE
        SET @sSetClause = @sSetClause + ',' + @sCRLF

    SET @sSetClause = @sSetClause + @sTAB + @sColumnName  + ' = '

    SET @sSetClause = @sSetClause + '@' + @sColumnName

 END

Next, we append a tab character, the column name to be updated, and an equals sign (=). For our Order_Details table, we would end up with this:

SET    UnitPrice = @UnitPrice,
       Quantity = @Quantity,
       Discount = @Discount

Next, we have the code that creates the WHERE clause for the new procedure. You will notice the code segment starts with an ELSE statement. This is the else condition of the primary key check—meaning we only run this code if the column is part of the primary key (see Figure 9).

Figure 9 Create the WHERE, if Necessary (Close Figure 9)
ELSE
 BEGIN
    IF (@sWhereClause = '')
        SET @sWhereClause = @sWhereClause + 'WHERE '
    ELSE
        SET @sWhereClause = @sWhereClause + ' AND '

    SET @sWhereClause = @sWhereClause + @sTAB + @sColumnName  + ' = @'
                       + @sColumnName + @sCRLF
 END

FETCH    NEXT
FROM     crKeyFields
INTO     @sColumnName, @nColumnID, @bPrimaryKeyColumn, @nAlternateType,
         @nColumnLength, @nColumnPrecision, @nColumnScale, @IsNullable,
         @IsIdentity, @sTypeName, @sDefaultValue
END

CLOSE crKeyFields
DEALLOCATE crKeyFields

Again, we either start the variable with the WHERE clause, or we append an AND clause, depending on whether or not it is the first item in the WHERE clause. Next, we append a tab character, the column name, the string " = @" and again, the column name. The results for our Order_Details example would look like this:

WHERE     OrderID = @OrderID
 AND     ProductID = @ProductID

Before ending the WHILE loop, we need to fetch the next row from the cursor and place the metadata values into our variables again. Once the loop completes, we close and de-allocate the cursor. Now we have all the information needed to print and optionally create our new run-time procedure (see Figure 10).

Figure 10 Print the Procedure (Close Figure 10)
SET     @sSetClause = @sSetClause + @sCRLF

SET     @sProcText = @sProcText + @sKeyFields + @sCRLF
SET     @sProcText = @sProcText + 'AS' + @sCRLF
SET     @sProcText = @sProcText + @sCRLF
SET     @sProcText = @sProcText + 'UPDATE    ' + @sTableName + @sCRLF
SET     @sProcText = @sProcText + @sSetClause
SET     @sProcText = @sProcText + @sWhereClause
SET     @sProcText = @sProcText + @sCRLF
IF @bExecute = 0
    SET     @sProcText = @sProcText + 'GO' + @sCRLF

PRINT @sProcText

IF @bExecute = 1
    EXEC (@sProcText)

Finally, the design-time stored procedure will print the T-SQL it created for the new run-time stored procedure by first adding a carriage return/line feed to the SET clause (for formatting purposes only). Next, we add our key fields (our procedure's parameters) and the keyword AS (required for a stored procedure's definition). Then, the UPDATE statement is appended, along with the name of the table that is to be updated. Finally, we append the SET clause variable and the WHERE clause variable, completing our procedure definition. As noted, the @sProcText variable that contains the T-SQL for the new run-time stored procedure can optionally be executed, and if executed, the new run-time stored procedure will be added to the database schema.

Conclusion

This is only one of the four automated stored procedures that we developed. Of course, each procedure will vary as required. For example, the design-time procedure that creates the run-time delete procedures only uses the primary key fields of each table, since that is all that is required to delete a row in any table. The source for all the user-defined functions and stored procedures is available for download at the link at the top of this article.

Many other features can be added to these procedures, some of which we mentioned already, such as the ability to accommodate bracketed object names, verification of object existence, creation of an audit trail, and use of an alter statement if the generated procedure already exists (great for maintaining security of the generated procedures and incorporating XML). You can also create additional settings in a table and use those to assist in the code generation. In other words, these procedures can be the starting point for other automated code generation tasks, or they may suit your needs as is. In either case, this code should help you save a lot of time and effort, and may even help you explore other interesting T-SQL techniques.

For related articles see:
.NET Reflection: Dynamically Bind Your Data Layer to Stored Procedures and SQL Commands Using .NET Metadata and Reflection
SQL and XML: Use XML to Invoke and Return Stored Procedures Over the Web
Serving the Web: Stored Procedure Wizard in Visual Basic Boosts Productivity

For background information see:
Inside Microsoft SQL Server 2000 by Kalen Delaney (Microsoft Press, 2000)
The Guru's Guide to SQL Server Stored Procedures, XML, and HTML by Ken Henderson (Addison-Wesley Professional, 2001)

Peter W. DeBettais a Wintellect trainer who consults and develops enterprise-level software solutions using Visual Basic, ASP, the .NET Framework, and SQL Server. Peter has coauthored several books, including SQL Server 7.0 Programming Unleashed (SAMS, 1999).

J. Byer Hillruns his own consulting company, Advanced Software Logic. Byer has been programming and designing databases for 11 years using Microsoft technologies such as Visual Basic, Visual C++, COM+, ADO, ASP, DHTML, and SQL Server. His primary focus is on application architecture, design, and efficient code reuse.


Data Access: Implement a Data Access Layer for Your App with ADO.NET

Data Access

Implement a Data Access Layer for Your App with ADO.NET

Dan Fox
This article assumes you're familiar with ADO.NET, Visual Basic .NET, and C#
Level of Difficulty 1 2 3
SUMMARY

Implementing data access functionality is a core activity of most developers working with the .NET Framework, and the data access layers they build are an essential part of their applications. This article outlines five ideas to consider when building a data access layer with Visual Studio .NET and the .NET Framework. The tips include taking advantage of object-oriented techniques and the .NET Framework infrastructure by using base classes, making classes easily inheritable by following guidelines, and carefully examining your needs before deciding on a presentation method and external interface.

Contents
Rule 1: Use Object-oriented Features
Rule 2: Adhere to the Design Guidelines
Rule 3: Take Advantage of the Infrastructure
Rule 4: Choose Your External Interface Carefully
Rule 5: Abstract .NET Framework Data Providers
Conclusion

I
f you're developing a data-centric application targeting the Microsoft® .NET Framework, you'll eventually need to create a data access layer (DAL). You probably know that there are benefits of building your code in the .NET Framework. Because it supports both implementation and interface inheritance, your code can be more reusable, especially by developers across your organization using different Framework-compliant languages. In this article, I'll present five rules for developing a DAL for your .NET Framework-based applications.

Before I begin, I should note that any DAL you build based on the rules discussed in this article will be compatible to the traditional multitier or n-tier application favored by developers on the Windows® platform. In this architecture, the presentation layer consists of Web Forms, Windows Forms, or XML Web Services code that makes calls to a business layer that coordinates the work of the data access layer. This layer consists of multiple data access classes. Alternatively, the presentation layer may make calls directly to the DAL in cases where business process coordination is not required. This architecture is a variant of the traditional Model-View-Controller (MVC) pattern and in many ways is assumed by Visual Studio® .NET and the controls that it exposes.

Rule 1: Use Object-oriented Features

The most fundamental object-oriented task is creating an abstract base class using implementation inheritance. This base class can include services that all your data access classes can use through inheritance. If the services are generic enough, they can be reused through the distribution of your base class throughout your organization. For example, in the simplest case, your base class can handle the creation of a connection object for your derived classes, as shown in Figure 1.

Figure 1 Simple Base Class (Close Figure 1)
Imports System.Data.SqlClient

Namespace ACME.Data

    Public MustInherit Class DALBase : Implements IDisposable

        Private _connection As SqlConnection

        Protected Sub New(ByVal connect As String)
            _connection = New SqlConnection(connect)
        End Sub

        Protected ReadOnly Property Connection() As SqlConnection
            Get
                Return _connection
            End Get
        End Property

        Public Sub Dispose() Implements IDisposable.Dispose
            _connection.Dispose()
        End Sub

    End Class

End Namespace

As you can see in this figure, the DALBase class is marked as MustInherit (abstract in C#) to ensure that it is used in an inheritance relationship. The class then includes a private SqlConnection object instantiated in the public constructor, which accepts the connection string as a parameter. The protected Connection property then allows derived classes to access the connection object, while the Dispose method from the IDisposable interface ensures that the connection object is disposed.

Even in the following simplified example you can begin to see the usefulness of abstract base classes:

Public Class WebData : Inherits DALBase

    Public Sub New()
         MyBase.New(ConfigurationSettings.AppSettings("ConnectString"))
     End Sub

     Public Function GetOrders() As DataSet

         Dim da As New SqlDataAdapter("usp_GetOrders", Me.Connection)
         da.SelectCommand.CommandType = CommandType.StoredProcedure
         Dim ds As New DataSet()

         da.Fill(ds)
         Return ds

     End Function

End Class
In this case, the WebData class inherits from DALBase and, as a result, does not need to worry about instantiating the SqlConnection object, but simply passes the connection string to the base class via the MyBase keyword (or the base keyword in C#). The WebData class's GetOrders method can access the protected property using Me.Connection (this.Connection in C#). While this example is relatively simple, as you'll see in Rules 2 and 3, the base class can provide other services as well.

An abstract base class is especially useful when your DAL needs to run in a COM+ environment. In this case, since the code required to allow the component to use COM+ is more complicated, it makes sense to create a serviced component base class like the one shown in Figure 2.

Figure 2 Serviced Component Base Class (Close Figure 2)
<ConstructionEnabled(True), _
Transaction(TransactionOption.Supported), _
EventTrackingEnabled(True)> _
Public MustInherit Class DALServicedBase : Inherits ServicedComponent

    Private _connection As SqlConnection

    Protected Overrides Sub Construct(ByVal s As String)
        _connection = New SqlConnection(s)
    End Sub

    Protected ReadOnly Property Connection() As SqlConnection
        Get
            Return _connection
        End Get
    End Property

End Class

In this code, the DALServicedBase class includes essentially the same functionality as in Figure 1, but additionally inherits from ServicedComponent in the System.EnterpriseServices namespace and includes attributes to indicate that the component supports object construction, transactions, and statistics tracking. The base class then takes care of catching the construction string configured in the Component Services Manager and once again creating and exposing the SqlConnection object. It's important to note that when a class inherits from DALServicedBase, it also inherits the settings for the attributes. In other words, a derived class will also have its transaction option set to Supported. If the derived class wants to override this behavior, it has the option of redefining the attribute at the class level.

In addition, your derived classes should avail themselves of both overloaded and shared methods where appropriate. There are essentially two scenarios where overloaded methods (a single method with multiple signatures) are used. First, they can be used when a method needs to accept arguments that vary in their types. The canonical examples of this type in the Framework are the methods of the System.Convert class. For example, the ToString method includes 18 overloaded methods that accept one argument, each of which is of a different type. Second, overloaded methods can be used to expose signatures with increasing numbers of arguments, but not necessarily arguments of different types. This type of overloading turns out to be very effective in a DAL since it can be used to expose alternate signatures for data retrieval and modification. For example, a GetOrders method can be overloaded so that one signature accepts no arguments and returns all orders, while an additional signature accepts an argument that indicates that the caller wants to retrieve only specific customer orders, as shown in the following code:

Public Overloads Function GetOrders() As DataSet
Public Overloads Function GetOrders(ByVal customerId As Integer) As
    DataSet
A good implementation tip in this case is to abstract the functionality of the GetOrders method into a private or protected method that can be called by each of the overloaded signatures.

Shared methods (static in C#) can also be used to expose fields, properties, and methods that are accessible to all instances of your data access classes. Although shared members cannot be used with classes that use Component Services, they can be useful for read-only data that can be retrieved in the shared constructor of your data access class and then read by all instances. Be careful when using shared members for read/write data since multiple threads of execution may be vying for access to the shared data.

Rule 2: Adhere to the Design Guidelines

In the online documentation that ships with Visual Studio .NET there is a topic titled "Design Guidelines for Class Library Developers", which covers the naming conventions for classes, properties, and methods in addition to patterns for overloaded members, constructors, and events that you should follow. One of the primary reasons you should follow the naming conventions is the cross-language inheritance provided by the .NET Framework. If you are building a DAL base class in Visual Basic® .NET, you'll want to ensure that developers using other languages compatible with the .NET Framework can inherit from it and easily understand how it works. By adhering to the guidelines I've outlined, your naming conventions and constructs won't be language specific. For example, you'll notice in the code samples in this article that camel casing (first word lowercased, plus intercaps) is used for the arguments to the methods, Pascal casing (each word uppercased) is used for the methods, and the base class has the Base suffix to denote that it is an abstract class.

A corollary to the .NET Framework design guidelines are the general design patterns like those documented in the book Design Patterns by the Gang of Four (Addison-Wesley, 1995). For example, the .NET Framework uses a variant of the Observer pattern called the Event pattern that you should follow when exposing events in your classes.

Rule 3: Take Advantage of the Infrastructure

The .NET Framework includes classes and constructs that can assist in handling generic infrastructure-related tasks such as instrumentation and exception handling. Combining these concepts with inheritance through your base class can be quite powerful. For example, consider the tracing functionality exposed in the System.Diagnostics namespace. In addition to the Trace and Debug classes, this namespace also includes classes derived from Switch and TraceListener. The Switch classes, BooleanSwitch and TraceSwitch, can be configured to be turned on and off both programmatically and through the application's configuration file. In the case of TraceSwitch, multiple levels of tracing can be exposed. The TraceListener classes, TextWriterTraceListener and EventLogTraceListener, direct the output of the Trace and Debug methods to a text file and event log, respectively.

As a result, you can add tracing functionality to your base class to make it simple for the derived classes to log messages. In turn, the application can control whether tracing is enabled using the application configuration file. You can add this functionality to the DALBase class shown in Figure 1 by including a private variable of type BooleanSwitch and instantiating it in the constructor:

Public Sub New(ByVal connect As String)
    _connection = New SqlConnection(connect)
    _dalSwitch = New BooleanSwitch("DAL", "Data Access Code")
End Sub

The arguments to BooleanSwitch include its name and description. You can then add a protected property to turn the switch on and off and a protected property to format and write a trace message using the WriteLineIf method of the Trace object:

Protected Property TracingEnabled() As Boolean
    Get
        Return _dalSwitch.Enabled
    End Get
    Set(ByVal Value As Boolean)
        _dalSwitch.Enabled = Value
    End Set
End Property

Protected Sub WriteTrace(ByVal message As String)
    Trace.WriteLineIf(Me.TracingEnabled, Now & ": " & message)
End Sub

In this way, derived classes don't have to know about the switch and listener classes themselves and can simply call the WriteTrace method when a significant event occurs in the data access class.

To create a listener and turn it on, the application configuration file can also be used. Figure 3 shows a simple configuration file that turns on the DAL switch just shown and directs its output to the file DALLog.txt through a TextWriterTraceListener called myListener. Of course, you can also programmatically create your own listener by deriving from the TraceListener class and including the listener directly in your DAL.

Figure 3 Tracing Configuration (Close Figure 3)
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
   <system.diagnostics>
       <switches>
       <add name="DAL" value="1" />
       </switches>
       <trace autoflush="true" indentsize="4">
          <listeners>
            <add name="myListener"
               type="System.Diagnostics.TextWriterTraceListener"
               initializeData="DALLog.txt" />
          </listeners>
       </trace>
   </system.diagnostics>
</configuration>

A second bit of infrastructure that you should certainly be taking advantage of is structured exception handling (SEH). At the most basic level, your DAL can expose its own Exception object inherited from System.ApplicationException and can additionally expose custom members. For example, the DALException object shown in Figure 4 can be used to wrap exceptions thrown by code in the data access classes. The base class can then expose a protected method to wrap the exception, populate the custom members, and throw it back to the caller, as follows:

Protected Sub ThrowDALException(ByVal message As String, _
  ByVal innerException As Exception)
    Dim newMine As New DALException(message, innerException)

    newMine.ConnectString = Me.Connection.ConnectionString
    Me.WriteTrace(message & "{" & innerException.Message & "}")
    Throw newMine
End Sub

Figure 4 Custom Exception Class (Close Figure 4)
Public Class DALException : Inherits ApplicationException

    Public Sub New()
        MyBase.New()
    End Sub

    Public Sub New(ByVal message As String)
        MyBase.New(message)
    End Sub

    Public Sub New(ByVal message As String, ByVal innerException As
                   Exception)
        MyBase.New(message, innerException)
    End Sub

    ' Add custom members here
    Public ConnectString As String
End Class

In this way, derived classes can simply call the protected method, passing in the data-specific exception (typically a SqlException or OleDbException) that was intercepted and adding a message that pertains to the particular data domain. The base class wraps the exception in a DALException and throws it back to the caller. This allows the caller to easily capture all exceptions from the DAL using a single Catch statement.

As an alternative, take a look at the Exception Management Application Block Overview published on MSDN®. This framework couples the publishing of exceptions with application logging through a set of objects. In fact, you can plug your own custom exception classes into this framework by inheriting them from the BaseApplicationException class provided in the .NET Framework.

Rule 4: Choose Your External Interface Carefully

As you design the methods of your data access classes, you need to consider how they will accept and return data. For most developers, there are three primary choices: using ADO.NET objects directly, using XML, and using custom classes.

If you're going to expose ADO.NET objects directly, you can employ one of two programming models. The first includes the DataSet and DataTable objects, which are useful for disconnected data access. Much has been written about the DataSet and its associated DataTables, but this is most useful when you need to work with data that's been disconnected from the underlying data store. In other words, a DataSet can be passed between the tiers of your application, even when the tiers are physically distributed, as is the case when the business and data services tiers are placed on a cluster of servers that are separate from the presentation services. Additionally, DataSet objects are ideal for returning data through XML-based Web Services because they're serializable and thus can be returned in a SOAP response message.

This is different from accessing data using classes that implement the IDataReader interface, such as SqlDataReader and OleDbDataReader. These data readers are used to access data in a forward-only, read-only manner. The big difference between the two is that the DataSet and DataTable objects can be passed between application domains (and thus between processes on the same or separate machines) by value, whereas the data reader can be passed around, but is always done so by reference. Take a look at Figure 5, in which Read and GetValues are executed in the server process and their return values copied to the client.

Figure 5 Remoting Data Readers
Figure 5 Remoting Data Readers

This diagram highlights how a data reader lives in the app domain in which it was created and all access to it results in a round-trip between the client and server application domain. This means that your data access methods should return data readers only when they are executing in the same app domain as the caller.

When using data readers, there are two other issues to consider. First, when you return a data reader from a method in your data access class, you need to think about the lifetime of the connection object associated with the data reader. By default, the connection remains busy while the caller is iterating through the data reader. Unfortunately, when the caller is finished, the connection remains open and is therefore not returned to a connection pool (if connection pooling is enabled). You can, however, instruct the data reader to close its connection when its Close method is called by passing the CommandBehavior.CloseConnection enumerated value to the ExecuteReader method of command objects.

Second, in order to decouple your presentation layer from a particular Framework data provider (such as SqlClient or OleDb), the calling code should reference the return value using the IDataReader interface rather than the concrete type, such as SqlDataReader. In this way, the presentation layer won't have to change if your application moves from an Oracle to a SQL Server™ back end and the return type of a method in the data access class changes.

If you want your data access classes to return XML, you can choose from the XmlDocument and XmlReader classes in the System.Xml namespace, which are analogous to the DataSet and IDataReader. In other words, your methods should return an XmlDocument (or XmlDataDocument) when the data is to be disconnected from its source, whereas XmlReader can be used for streamed access to XML data.

Finally, you can also decide to return custom classes with public properties. These classes can be marked with the Serialization attribute so they can be copied across application domains. Plus, you'll likely need strongly typed collection classes if you want to return multiple objects from your methods.

Figure 6 contains an example of a simple Book class and its associated collection class. You'll notice that the Book class is marked with Serializable to enable "by value" semantics across application domains. This class implements the IComparable interface so that when it is included in a collection class, it will be sorted by Title by default. The BookCollection class derives from ArrayList in the System.Collections namespace and shadows the Item property and Add method in order to restrict the collection to only Book objects.

Figure 6 Using a Custom Class (Close Figure 6)
Imports System.Xml.Serialization

<Serializable()> _
Public Class Book : Implements IComparable

    <XmlAttributeAttribute()> Public ProductID As Integer
    Public ISBN As String
    Public Title As String
    Public Author As String
    Public UnitCost As Decimal
    Public Description As String
    Public PubDate As Date

    Public Function CompareTo(ByVal o As Object) As Integer _
      Implements IComparable.CompareTo
        Dim b As Book = CType(o, Book)
        Return Me.Title.CompareTo(b.Title)
    End Function

End Class

Public NotInheritable Class BookCollection : Inherits ArrayList

    Default Public Shadows Property Item(ByVal productId As Integer) _
      As Book
        Get
            Return Me(IndexOf(productId))
        End Get
        Set(ByVal Value As Book)
            Me(IndexOf(productId)) = Value
        End Set
    End Property

    Public Overloads Function Contains(ByVal productId As Integer) As _
      Boolean
        Return (-1 <> IndexOf(productId))
    End Function

    Public Overloads Function IndexOf(ByVal productId As Integer) As _
      Integer
        Dim index As Integer = 0
        Dim item As Book

        For Each item In Me
            If item.ProductID = productId Then
                Return index
            End If
            index = index + 1
        Next
        Return -1
    End Function

    Public Overloads Sub RemoveAt(ByVal productId As Integer)
        RemoveAt(IndexOf(productId))
    End Sub

    Public Shadows Function Add(ByVal value As Book) As Integer
        Return MyBase.Add(value)
    End Function

End Class

By using custom classes you get complete control of how the data is represented, developer productivity through strong typing and IntelliSense®, and no caller dependency on ADO.NET. However, this approach requires more code since the .NET Framework does not contain any object-relational mapping technology (other than typed DataSet objects that are essentially derived DataSet classes). In these cases, you'll typically create a data reader in your data access class and use it to populate the custom class.

Rule 5: Abstract .NET Framework Data Providers

The final rule specifies why and how you should abstract the .NET Framework data provider used internally in your DAL. As I've mentioned, the ADO.NET programming model exposes distinct .NET Framework data providers including SqlClient, OleDb, and others available on the MSDN Online Web site. While this design results in improved performance and the ability for providers to expose data-source-specific functionality (such as the ExecuteXmlReader method of the SqlCommand object), it forces you to decide which provider to code against. In other words, a developer typically chooses to use SqlClient or OleDb and then writes code directly against the classes in the respective namespace.

If you want to change the .NET Framework data provider, you'll need to recode your data access methods. To avoid this, you can use a design pattern known as the Abstract Factory. Using this pattern, you can build a simple class that exposes methods to create the primary .NET Framework data provider objects (command, connection, data adapter, and parameter) based on information identifying the .NET Framework data provider passed into the constructor. The code in Figure 7 shows a simple C# version of this class.

Figure 7 ProviderFactory (Close Figure 7)
public enum ProviderType :int {SqlClient = 0, OLEDB = 1}

public class ProviderFactory {

    public ProviderFactory(ProviderType provider) {
        _pType = provider;
        _initClass();
    }

    public ProviderFactory() {
        _initClass();
    }

    private ProviderType _pType = ProviderType.SqlClient;
    private bool _pTypeSet = false;
    private Type[] _conType, _comType, _parmType, _daType;

    private void _initClass() {
    _conType = new Type[2];
    _comType = new Type[2];
    _parmType = new Type[2];
    _daType = new Type[2];

       // Initialize the types for the providers
       _conType[(int)ProviderType.SqlClient] = typeof(SqlConnection);
       _conType[(int)ProviderType.OLEDB] = typeof(OleDbConnection);
       _comType[(int)ProviderType.SqlClient] = typeof(SqlCommand);
       _comType[(int)ProviderType.OLEDB] = typeof(OleDbCommand);
       _parmType[(int)ProviderType.SqlClient] = typeof(SqlParameter);
       _parmType[(int)ProviderType.OLEDB] = typeof(OleDbParameter);
       _daType[(int)ProviderType.SqlClient] = typeof(SqlDataAdapter);
       _daType[(int)ProviderType.OLEDB] = typeof(OleDbDataAdapter);
    }

    public ProviderType Provider {
        get {
            return _pType;
        }
        set {
           if (_pTypeSet) {
                throw new ReadOnlyException("Provider already set to "
                    + _pType.ToString());
            }
            else {
                _pType = value;
                _pTypeSet = true;
            }
        }
    }
    public IDataAdapter CreateDataAdapter(string commandText,IDbConnection
                                          connection) {
        IDataAdapter d;
        IDbDataAdapter da;

        d = (IDataAdapter)Activator.CreateInstance(_daType[(int)_pType],
                                                   false);
        da = (IDbDataAdapter)d;
        da.SelectCommand = this.CreateCommand(commandText, connection);
        return d; }

    public IDataParameter CreateParameter(string paramName, DbType
                                          paramType) {
        IDataParameter p;
        p = (IDataParameter)Activator.CreateInstance(_parmType[(int)_pType],
                                                    false);
        p.ParameterName = paramName;
        p.DbType = paramType;
        return p;
    }

    public IDataParameter CreateParameter(string paramName, DbType
                                          paramType, Object value) {
        IDataParameter p;
        p = (IDataParameter)Activator.CreateInstance(_parmType[(int)_pType],
                                                    false);
        p.ParameterName = paramName;
        p.DbType = paramType;
        p.Value = value;
        return p;
    }

    public IDbConnection CreateConnection(string  connect) {
        IDbConnection c;
        c = (IDbConnection)Activator.CreateInstance(_conType[(int)_pType],
                                                    false);
        c.ConnectionString = connect;
        return c;
    }

    public IDbCommand CreateCommand(string cmdText, IDbConnection
                                    connection) {
        IDbCommand c;
        c = (IDbCommand)Activator.CreateInstance(_comType[(int)_pType],
                                                 false);
        c.CommandText = cmdText;
        c.Connection = connection;
        return c;
    }
}

In order to use this class, the code in your data access classes would need to program against the various interfaces that the .NET Framework data providers implement, including IDbCommand, IDbConnection, IDataAdapter, and IDataParameter. For example, in order to fill a DataSet with results from a parameterized stored procedure, you could use the following code inside a method of your data access class:

Dim _pf  As New ProviderFactory(ProviderType.SqlClient)
Dim cn As IDbConnection = _pf.CreateConnection(_connect)
Dim da As IDataAdapter = _pf.CreateDataAdapter("usp_GetBook", cn)

Dim db As IDbDataAdapter = CType(da, IDbDataAdapter)
db.SelectCommand.CommandType = CommandType.StoredProcedure
db.SelectCommand.Parameters.Add(_pf.CreateParameter("@productId", _
    DbType.Int32, id))

Dim ds As New DataSet("Books")
da.Fill(ds)

Typically, you would declare the ProviderFactory variable at the class level and instantiate it in the constructor of the data access class. Additionally, its constructor would be populated with the provider read from a configuration file, rather than hardcoded, as shown here. As you can imagine, the ProviderFactory would be a great addition to your DAL base class and can then be included in the assembly and distributed to other developers.

You can choose to take it a step further and encapsulate common ADO.NET code that developers write over and over. In fact, Microsoft has released a Data Access Application Block that performs this function for SQL Server (see Data Access Application Block Overview).

Conclusion

In the era of Web Services, more and more applications will be built to manipulate data from an isolated application layer. If you follow some basic rules and commit them to habit, writing data-access code will be faster, easier, and more reusable, save you trips to the server, and allow you to keep your data separate.

For related articles see:
ADO.NET - Data Access in .NET
Data Access Application Block Overview

For background information see:
Design Patterns by Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, (Addison-Wesley, 1995)
Teach Yourself ADO.NET in 21 Days by Dan Fox (SAMS, 2002)
Exception Management Application Block Overview

Dan Foxis a Technical Director for Quilogy (http://www.quilogy.com) in Overland Park, Kansas where he evangelizes technology. Dan is the author of Teach Yourself ADO.NET in 21 Days and Building Distributed Applications with Visual Basic .NET (both from SAMS). Dan can be reached at dfox@quilogy.com.


Editor's Note: MP3 Playlists with Visual Basic .NET

Editor's Note
MP3 Playlists with Visual Basic .NET

Code download available at: EditorsNote0304.exe (132KB)
Contents
MP3 Playlist Editor

Although it's been several months since the slammer virus hit SQL Server installations, we'd like to offer a public service announcement for anyone who hasn't patched their machine yet. PATCH YOUR MACHINE NOW! We found out the hard way that it can hit any machine that runs SQL Server and is attached to the network. You might not notice any symptoms at first—we thought that our network router had gone bad because no one could get a lease from the DHCP feature on it. It turns out that the outgoing network storm from the one infected machine was tying up all networking to such an extent that the router couldn't get a word in edgewise. If you start to have weird network problems, take machines offline one by one until they magically go away. Or check Task Manager—if sqlservr.exe is taking up more CPU than the System Idle Process, it might have been compromised.

An even simpler solution is to visit the Microsoft SQL Server site. In addition to the SQL Server SP3 patch that fixes this vulnerability, the product support team has created a program that will scan your network for unsecured instances of SQL Server and tell you where they are. It's up to you to stop SQL Server services on that machine and then patch it appropriately. Patching production products might be a nuisance, but it sure beats the alternative.

MP3 Playlist Editor

For the second part of this month's Editor's Note, we refer you first to a communication we had from an old acquaintance not long ago. "Hey Ed.," the note began, "remember me? Chris! We met at that Coast Guard dance a few years ago. Listen, I just got the same MP3 player you did, but I don't know how to make playlists for it. Got any ideas?"

Leaving aside for a minute the fact that we've never been to any Coast Guard dance and we have no idea how Chris knows we own the same 20GB MP3 portable player as he does, this note got us to thinking. It couldn't be that hard to write one of these ourselves. After all, a playlist (commonly saved with an .m3u extension) is just a plain text file containing paths to MP3 files.

It would be easy to write a program that looped through a path recursively, searching out all files with an .mp3 extension, then letting the user select any or all of them for a list that then got written out to a file. The System.IO.Directory namespace of the .NET Framework provides a GetDirectories member function that places all subdirectory names in a string array (which can then be recursed in a function), and GetFiles does the same thing for all the file names you need to look at.

But MP3 files contain a lot of tagged data, from song and album name to artist to track number. There are several dozen possible tags (listed at http://www.id3.org/id3v2.3.0.txt). While most of these tags aren't used in the majority of MP3 files, one tag quickly caught our eye—APIC, which represents the embedded cover image of the album the song is found on. This got us to thinking how we could doll up the program even further.

It just so happens that the data found after this tag provides length information, a couple of flags, and a stream of data representing the image (usually JPEG format). You just read the data out of the file and into an array of bytes, and...what? Before the .NET Framework, it would have been a hassle to get this data into some sort of image or picture control. But now anyone with Visual Basic .NET and five minutes on their hands can do it. All you have to do is take the array of bytes, create a MemoryStream from it, and create an Image using the FromStream method. Thanks to the .NET Framework, you can take an array of bytes holding JPG data and display it as an image in a PictureBox control in as little as one simple line of code, as shown here:

picCover.Image = Image.FromStream(New MemoryStream(jpgdata))
How cool is that? If you use only one .NET Framework trick this year, make it the direct memory bitmap copy one.

We continue to be the only Editor's Note with a code download, and the full source code for this MP3 Playlist Maker, an application in progress, can be downloaded from the link at the top of this article. We've just added a ListView, so that you can create a playlist by song name instead of by file name. And it works great on these high-capacity MP3 players! If you have any tweaks or additions to this program, find it useful, or have any features you might like to see implemented in a future version of the Editor's Note, please feel free to drop us a line at mmeditor@microsoft.com.

—J.T.

Active Directory, ActiveX, IntelliSense, Microsoft, MSDN, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Generate Bar Codes for Microsoft Access
Agentless Availability Monitoring
Print from .NET Framework Apps
64-bit Support in ctree
Model Data and Generate C# Code
Enterprise Data Backup and Synchronization
Data Warehousing Tools
The Bookshelf

Generate Bar Codes for Microsoft Access

TALtech has announced a new version of its Bar Code ActiveX Control for Microsoft® Access, Excel, and Word. This new limited license version is based on TALtech's Bar Code ActiveX® PLUS version, limited for use on up to five workstations. The new Bar Code ActiveX LL (Limited License) will be useful for small organizations that want to print bar codes on labels or documents from their Microsoft Office applications. The control supports data binding, allowing the user to incorporate bar-coding into database reporting and labeling applications.

Agentless Availability Monitoring

MKS Inc. released MKS AlertCentre 2.0, an agentless monitoring solution for Windows® which includes monitoring wizards for Microsoft Internet Information Services, SQL Server™, and Exchange. AlertCentre 2.0 allows you to monitor unlimited devices and applications from each licensed monitoring station with no limits on the number of users who can access AlertCentre either locally or remotely. Its agentless architecture means there is no software to deploy to the devices you're monitoring.

New features in MKS AlertCentre 2.0 include application monitoring wizards which automatically discover infrastructure and application elements for supported servers, and automatic generation of system and application monitors necessary to comprehensively manage them for availability and performance.

Also included is a Web-based, at-a-glance status-of-operations console with point-and-click drilldown, and toolbar and status frames. AlertCentre 2.0 carries this highly functional, easy interface theme throughout.

There are new monitor types for SNMP devices, UDP ports, incoming and outgoing e-mail services, printers/copiers, files, directories, machine shares, SSL certificates, newsgroups, Web site link integrity, and database queries.

MKS also includes the ability to control when and how actions can be triggered, run pre- and post-action commands or programs, and start, stop, and restart services. Reports include uptime, status of operations, monitor log, and configuration.

Print from .NET Framework Apps

PrintForm for .NET, recently released by TMG Development Ltd., helps you add printing support to an application by printing the Windows Forms you have already designed and coded using Visual Studio® .NET. Reports and printouts can now look the way you want them to look. PrintForm is designed to simplify the process of adding custom printing to Microsoft .NET Framework-based applications, which is a lengthy task involving graphics (GDI+) calls to accurately render their printed output. Key product features include the ability to print the contents of any Windows Form or other container control either directly to the printer or to the PrintPreviewDialog and integration with Visual Studio .NET and the Framework printing architecture. PrintForm is derived from the PrintDocument class in the Framework and plugs directly into the PrintPreviewDialog classes supplied with the Framework. Multipage reports are produced either as repetitions of the same page layout with different data or as entirely different layouts on each page.

PrintForm can render all Windows Forms controls and has support for the majority of controls which use legacy technology (such as those from the Windows common controls DLL—ListView, TreeView, DatePicker, and so on). Because PrintForm can render any control, printouts can contain dynamically created charts, images, and custom user controls.

The program also provides header and footer support and supports printing form controls with an alternative background (for example controls which display on screen with a grey background can be printed with a white background). PrintForm is a fully managed C# component, compatible with all common language runtime-targeted languages and requires the CLR version 1 or above.

64-bit Support in ctree

FairCom Corporation announced that their c-tree Plus embedded database development system and c-tree database Server now support Windows XP 64-bit Edition. This new technology targets customers working in scientific and engineering fields and other industries which involve managing large amounts of complex data.

c-tree Plus includes development servers for all supported platforms (Windows, Macintosh, Unix, and more) as well as free development ODBC and native Seagate Crystal Report drivers.

Model Data and Generate C# Code

Altova Inc. has announced XMLSPY 5 Release 3, which adds developer support for various widely used enterprise technologies, including apps built using the .NET Framework. The XMLSPY 5 Code Generator now supports C# code generation. You can use XMLSPY to model data elements in XML schema, then XMLSPY can auto-generate C# classes (data bindings) corresponding to elements defined in the data model. The generated code uses System.XML, the new .NET Framework API for programmatically accessing XML documents.

The upgrade also includes new data editing and server administration features for Oracle XML DB. XMLSPY's new features enable you to perform common operations on data managed by XML DB.

In addition, the program includes a Web Service Description Language (WSDL) documentation generation utility, allowing you to document and publish a Web Service interface. Any WSDL file can be annotated and published into a Microsoft Word or HTML output file. The stylesheet designer supports visual editing and generation of XSL Formatting Object code, which enables XML content to be rendered into a PDF file.

Enterprise Data Backup and Synchronization

Peer Software Inc. recently released PeerSync Pro 7.0, an enterprise-wide data backup and synchronization solution. The software's data routing technology can be programmed to adapt to circumstances that can compromise data availability and consistency; these include system failure, the inability to access locked or open files, and the out-of-sync data of an organization's mobile workforce.

PeerSync 7.0 monitors connections between storage points (for example, server to server or desktop to desktop). If a connection is down, the software will seek an alternative data path or wait until the connection is resumed to perform the backup/synchronization operation. The new version also incorporates Embedded Open File Manager software developed by St. Bernard Software, which lets you protect files even while they are locked or in use.

In addition to using PeerSync to back up and synchronize standard business documents, users can continue working without interruption as open files are unobtrusively synchronized. New features include the ability to monitor target and source devices for connectivity, and suspend activity until a connection is resumed. This allows PeerSync to ensure that synchronizations are only attempted when a connection exists. Users now have alternate or additional target paths for their data.

The Enhanced Open File Manager feature allows back up and synchronization of open or locked files, including system files and databases. All files that were not correctly synchronized—whether the user is in real-time, scheduled, or interval mode—can be retried.

Data Warehousing Tools

CONNX Solutions has announced CONNX 8.8, the latest release of its data access middleware product. Direct support for Microsoft .NET technology has been added to CONNX 8.8 with the introduction of a pure CONNX OLE DB provider. Users will be able to achieve the performance of a native provider while writing their own applications in managed C# or Visual Basic® .NET code.

The new release also includes the CONNX Data Synchronization Tool, which allows users to move enterprise data from any source location to any target data source. This Windows-based application employs hash keys to quickly detect when records are updated, deleted, or inserted. By storing these hash keys, along with the primary key of each record, in a separate CONNX private data store, the CONNX Data Synchronization tool performs incremental updates from source to target at a fraction of the usual update speed.

Once the target source is populated, CONNX can be used with other tools for ad hoc reporting, application development, and Web development. This engine allows querying either against the data warehouse and data marts or against the live operational data sources.

The Bookshelf

O'Reilly recently published The XSLT Cookbook by Sal Mangano, which contains hundreds of solutions to common problems confronted by XSLT developers. Each recipe walks through a problem and its solution, with explanations of the choices made and techniques used in creating that solution. Some of the topics covered are string manipulation, mathematical processing, and date-and-time handling; interactions between calendar systems; selecting content in source documents converting from XML to plain text; efficient tree manipulation; and XML document tweaking.

You'll also find information regarding using XSLT to query XML documents, generating HTML, C, and XSLT code using XSLT, and creating charts and graphs with SVG and XSLT. For more information about the book see http://www.oreilly.com/catalog/xsltckbk.

Contacts

Altova Inc. 900 Cummings Center, Suite 314 T, Beverly, MA 01915, 978-816-1600, http://www.altova.com

CONNX Solutions 1800 - 112th Avenue NE, Suite 150, Bellevue, WA 98004, 888-882-6669, 425-519-6600, http://www.connx.com

FairCom Corporation 2100 Forum Blvd. Suite C, Columbia, MO 65203-5456, 800-234-8180, 573-445-6833, http://www.faircom.com

MKS Inc. 410 Albert Street, Waterloo, ON Canada N2L 3V3, 800-265-2797, 519-884-2251, http://www.mks.com

Peer Software 1363-26 Veterans Highway, Hauppauge, NY 11788, 631-979-1770, http://www.peersoftware.com

TALtech 2027 Wallace Street, Philadelphia, PA 19130, 800-722-6004, http://www.taltech.com

TMG Development Ltd. http://www.winformreports.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Careyis a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: Consuming a DataSet in ASP.NET, XML and SQL, and More

Web Q&A
Consuming a DataSet in ASP.NET, XML and SQL, and More
Edited by Nancy Michell

Q How can a SQLXML Web Service-returned .NET Framework DataSet be consumed in ASP.NET?

A Microsoft® SQLXML is an ISAPI extension for Microsoft Internet Information Services (IIS) which maps HTTP requests for XML data to SQL Server™ 2000 (see http://msdn.microsoft.com/sqlxml). With SQLXML 3.0, stored procedures can be exposed as .NET Framework DataSets returning XML-based Web Services. To do so, you'll want to bind these DataSets in an ASP.NET Web form. Here you'll learn how to use the SQLXML user interface to expose a stored procedure as a Web Service, add a Web reference in Visual Studio® .NET to the Web Service, bind the resulting DataSet to an ASP.NET DataGrid, and show just how easy it is to do all this.

In order to get started on your server, you will need to be running IIS on Windows® 2000, Windows XP Professional, or Windows Server 2003; SQL Server 2000; SQLXML 3.0 SP1 (which you can download from the SQLXML Web site at http://msdn.microsoft.com/sqlxml); and SOAP Toolkit 2.0 SP2 (which you can download from the SOAP Web site at http://msdn.microsoft.com/SOAP).

In this example, a single stored procedure from the Northwind sample database will be exposed.

To begin, click on Start | Programs | SQLXML 3.0 | Configure IIS Support to open the IIS Virtual Directory Management for SQLXML 3.0. Expand the MMC snap-in to the Default Web site node, right-click, and select New | Virtual Directory to open the New Virtual Directory properties page.

Next, name the new virtual directory "NorthwindWebSvc" and point it to C:\NorthwindWebSvc or another folder. From the Security tab, select "Use Windows Integrated Authentication" and then from the Data Source tab, select the Northwind database. Because SOAP must post its request to the XML Web Service, make sure "Allow Post" is checked in the Settings tab.

Next, from the Virtual Names tab, type "SOAP" in the name textbox and select SOAP from the type dropdown. Then you should point the path to C:\NorthwindWebSvc\SOAP\ or another directory. The directory's Access Control List can be used to restrict access to the Web Service.

Finally, click Save and then Configure (see Figure 1), then click the [...] button to select a stored procedure. You can select SalesByCategory. The method name will be generated based on the stored procedure name. Make sure the "Output as" option has Single dataset selected, then click Save, and then click OK.

The Northwind XML Web Service is now configured and has one available method—SalesByCategory. In the next step, you'll bind the returned dataset to a DataGrid in an ASP.NET Web Form.

Figure 1 Exposing a Stored Procedure
Figure 1 Exposing a Stored Procedure

At this point you need to add a Web Reference in a Visual Studio .NET Web Forms Project. Once you have added the reference, Visual Studio .NET will generate a proxy class to all available methods in your SQXML Web Service. First, open a new Visual Studio .NET ASP.NET Web Forms Project in your language of choice. Then from the Solution Explorer of your new project, expand to the references node, right-click, and select Add Web Reference (see Figure 2). In the Add Web Reference Dialog enter http://localhost/NorthwindWebSvc/SOAP?wsdl into the Address field and then click Add Reference.

Figure 2 Add Web Reference
Figure 2 Add Web Reference

Now you'll need to bind to the ASP.NET Web Form DataGrid. Open Webform1.aspx, which was automatically generated when you created the new project. Drag and position a new DataGrid anywhere in the Web form. Then double-click in the Web form to open the codebehind class, locate the Page_Load routine, and paste in the following code:

int outParam = 0;
localhost.SOAP oService = new localhost.SOAP();

oService.Credentials = System.Net.CredentialCache.DefaultCredentials;

DataSet myDS=oService.SalesByCategory("Produce","2000",out outParam);
DataGrid1.DataSource = myDS;
DataGrid1.DataBind();
Save WebForm1.aspx and open Web.config. Add the following identity element with impersonate set to true so that ASP.NET can pass on the current users credentials to SQL Server:
<identity impersonate="true" />
Finally, build and browse WebForm1.aspx to enjoy the results.

Q I have a really large database and I want to select specific data from it. I am considering the following options: I can make views and select the data I need from them, I can use stored procedures, or I can talk directly to the tables or to the views. Is there another way I can speed up data retrieval on the client?

A Well, if you don't need the return set all at once—for example, if you're paging through it on a Web page—you have a couple of options available to you.

First, you could use a unique key to keep track of your location in the table. Do something like the following:

SELECT TOP 50 * FROM foo WHERE pk>@lastkey ORDER BY pk
Here, 50 is the number of rows you want to display on each page. You retrieve @lastkey from the final row of each set returned. You could also return header- or summary-level data, then allow the user to drill down into it. For example, instead of returning all the details for all orders in an orders table, try to just return the header info and provide the functionality to further drill into an individual order. That can greatly reduce the amount of data you have to return to the client in one shot and thereby speed it up while still providing the option to get more info on the items of interest.

Q Is there COM support for SQLXML Managed Classes? If so, how is it implemented? In other words, are COM wrappers built for a Microsoft .NET Framework implementation, or vice versa?

A The SQLXML managed provider is a wrapper around SQLXMLOLEDB. So use the SQLXMLOLEDB provider with ADO for COM. SQLXML Bulk Load only provides a COM interface.

All other SQLXML functionality that is not SOAP-based is included in the ADO Stream/ICommandStream object model via COM. The SQLXMLOLEDB provider just creates the XML on the client side, gives variations to the FOR XML modes, and allows pass-through of a recordset returning a stored procedure.

SOAPSQL requires a virtual directory and is not exposed directly for a COM call. SQLXML SOAP is only implemented over HTTP. But the configuration tool to set up the virtual directories and the SOAP method mappings is fully automated through COM Automation. Take a look at the SQLXML 3.0 docs (http://msdn.microsoft.com/sqlxml) for more information.

Q I have an XML file that will not fit into a SQL varchar field. How can I load it? I see that sp_xml_preparedocument can take ntext as a parameter, but I cannot figure out how to do that.

Suppose my XML data is in an ntext column in a table, like this:

SELECT Text FROM XmlData WHERE ID=1
How do I get that into sp_xml_preparedocument so that I can parse and use the XML in an OPENXML statement?

A You can pass the XML file content to the server as an ntext parameter to a stored procedure that then uses sp_xml_preparedocument and OpenXML in order to extract the information into the relational tables.

Unfortunately, SQL Server 2000 has no easy way to get an ntext value from a table into a parameter, regardless of XML. There are two other workarounds: use nested EXECs or use the OLE Automation stored procs to externalize and reimport the data via OLE DB.

Q If I create a SQLXML virtual directory in IIS (with sqlxmlx.dll 1.0), set the authentication with SQL Server using a SQL user, and allow template queries, then under the IIS MMC snap-in set the authentication mode for the virtual directory as "Windows Authentication," the NTFS security for template XML files gives everyone access. If I log onto a machine with a local machine user account and execute the template query, I expect an "access denied" error, but it didn't fail. Why not?

A When SQLXML is configured to use SQL Server authentication, it sets the IIS authentication to be anonymous. When you set the IIS authentication type to Windows, make sure that the anonymous authentication is turned off; otherwise the Windows authentication doesn't have much effect. Since template file access is not restricted, it does not matter how you log in.

Q I need some advice on how to implement the following scenario. I want to use SQL Server in a clustered configuration with a Storage Area Network (SAN) back end. Also, I want to use an active/passive configuration where the passive node is to build online analytical processing (OLAP) cubes from the data on the active node while in normal operation. I want to use split backup on the SAN side to create a separate read-only mirror, then mount the split mirror on the passive node as a separate volume to conduct online analytical processing.

Can this be done? If so, how? Finally, are there any problems related to this solution?

A OLAP is not cluster aware. It can be made a generic cluster service in a server cluster, but it's not the same as the default SQL Server 2000 failover clustering. See Knowledge Base article 308023 for more information.

Also, you are describing two instances; by your definition, it would be active/active. You can use Network Load Balancing (NLB) to make Analysis Services available, too (see Creating Large-Scale, Highly Available OLAP Sites). One issue that you didn't mention is how the OLAP cubes are being queried. That must be done from the same virtual server that is doing the OLAP cube processing. What you cannot do (which is what everyone would like to do, but can't) is to have one node in the cluster doing partition processing while another is performing queries. The OLAP data folder can only be accessed by one server in the cluster at a time.

There is no problem with doing this:

NodeA (SQL active/OLAP passive): the RDBMS
NodeB (SQL passive/OLAP active): the OLAP cubes
In effect, each system is providing the backup of the other.

That will work very well and if you follow the instructions for the Analysis Services setup in the previously cited Knowledge Base article, you will have no problem. You could replicate the data from the "active" node to the other node (another instance that is not clustered) and build cubes on the nonclustered instance. Then the OLAP piece wouldn't failover, but the Online Transaction Processing (OLTP) piece would.

However, it is probably (for now at least) better to put OLAP on nonclustered boxes, use NLB to make it available, and cluster your main data store, which is your OLTP repository. Otherwise, more would need to be known about the whole architecture.

Q I read that SQL Server 2000 always tries to register a virtual server in Active Directory® first, then it tries Windows NT® LAN Manager (NTLM). (See Knowledge Base article 303411.) Does this mean that when you failover a SQL Server 2000 server in a Windows NT domain it takes longer because you can't talk to the Active Directory controller? I noticed the failover on SQL Server 2000 in my production environment was a lot slower than my live one which is authenticating to Active Directory. Does SQL attempt to re-register upon failover? What would be some reasons that failover would take longer in one environment than another, all other things (hardware, network, and so on) being equal?

A Upon failover, SQL Server stops on one node and starts on the other. Upon startup, SQL Server will attempt to register a Service Principal Name (SPN) in the Active Directory. If this fails because of the reasons in the Knowledge Base article you cited, it doesn't mean that SQL Server can't talk with a domain controller. If that were the case, SQL Server might not be able to start at all because the SQL Server startup account couldn't authenticate with the domain. One thing that could affect failover time is the length of recovery when SQL Server starts up, such as transactions needing to be rolled forward or back.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers: Avner Aharoni, Cihan Biyikoglu, Michael Brundage, Srinivasa Burugapalli, Stefan Chudoba, Bruno Denuit, Murat Erentürk, William French, Max Giolitti, Matthew Goldberg, Brian Goldstein, Doug Harrison, Ken Henderson, Allan Hirt, Doug Hoch, Quentin Hurd, David Ikeda, Julia Jia, Bhalinder Kehal, Sumit Lahoti, Jinghao Liu, Farooq Mahmud, Andrew Minkin, Josh Mitts (Volt), Amar Nalla, Michael Rys, Ajith Vaithianathan, Mark Westmoreland, Dave Wickert.


Cutting Edge: MyTracer Monitors and Traces ASP.NET Apps

Cutting Edge
MyTracer Monitors and Traces ASP.NET Apps
Dino Esposito

Code download available at: CuttingEdge0304.exe (187KB)
Contents
Trace-enabled ASP.NET Pages
Emitting Trace Messages
The Trace Viewer
The MyTracer Tool
Enabling MyTracer
The Helper Web Service
The Windows-based Application
Putting It All Together

The Microsoft® .NET Framework comes with a rich set of programming tools for debugging and tracing applications. I'm not talking about integrated debuggers; I'm referring to software components that you use in the development cycle. Debuggers are a different animal because they are interactive tools that normally integrate with a full-fledged IDE such as Visual Studio® .NET.

The Systems.Diagnostics namespace defines two classes, named Trace and Debug, whose methods can be used to output messages containing information about code execution. This feature is extremely useful for tracking data inconsistencies, monitoring execution flow and assert conditions, and even gathering profiling data. The Trace and Debug classes are nearly identical and work on top of more specialized modules called listeners.

The tools you use for tracing Windows® and ASP.NET applications have rather different characteristics. In particular, the tracing mechanism supported by Windows Forms applications has an extensibility model based on custom listeners; the trace mechanism for ASP.NET applications does not. Instead, the ASP.NET subsystem provides a custom tracing model. When a page executes in trace mode, a variety of tables are appended to the output and they display information about performance, requests, and state. Regardless of whether you write messages to the page's trace log, those tables are always appended at the bottom of the output sent to the browser.

Visual Studio .NET provides a powerful, integrated debugger that supports cutting-edge features such as breakpoints, quick watch, and various ways of stepping through the code. The debugger is the ideal tool to catch errors and step into small but critical portions of the code. So if you use Visual Studio .NET to build your project, you can't ask for more. But what if you don't use Visual Studio .NET? What if you use an editor that doesn't provide debugging support? In such situations, the importance of programming tools for tracing the behavior of an application grows considerably.

In this column, I'll first examine the tracing subsystem available to all ASP.NET applications and then write a Windows Forms application that executes a given page and collects runtime information such as the page's view state, the request context, and the contents of global objects like Cache, Application, and Session. The application, which I've called MyTracer, can be integrated with Visual Studio .NET and constitutes an alternative to the default browser, at least during the testing phase. I should point out that in certain places I'll be using undocumented classes. It's unlikely to affect the code, but those classes are subject to future changes.

Trace-enabled ASP.NET Pages

Although a trace configuration section can be added to the web.config file to configure tracing at the application level, tracing is a feature that you often control on a per-page basis. For large projects, you can toggle the tracing attribute on and off using the following code in the application's web.config file:

<configuration>
  <system.web>
     <trace enabled="true" pageOutput="true" />
  </system.web>
</configuration>
The enabled attribute authorizes tracing on the application, while the pageOutput attribute permits output to appear in the page. If pageOutput is set to false (which is the default setting), then the tracing output is automatically routed to the ASP.NET tracer tool, trace.axd. At the end of the project, you simply set both of the trace element's attributes to false. This way, you eliminate the risk that you'll inadvertently leave tracing enabled.

To enable tracing only on a particular page, you use the trace attribute in the @Page directive. The default value for the attribute is false. When you set it to true, both system and custom tracing information appear at the bottom of the page, as shown in Figure 1.

Figure 1 System and Custom Tracing
Figure 1 System and Custom Tracing

You should note that the trace information is part of the actual page output and, as such, displays through any type of browser that accesses the page. As you can guess from the figure, some tables of information show up along with the trace information generated by the page. Additional tables display request details, the control tree, and some useful collections such as cookies, headers, form values, and server variables. If the session and the application state are not empty, the contents of the Session and Application intrinsic objects are also included in the view. The contents of the view state and the Cache object are not flushed to the trace log. These are the key enhancements that my tool will provide.

The @Page directive also supplies the TraceMode attribute to let you choose the order in which the information should be displayed. Feasible values are the strings SortByCategory and SortByTime. By default, the trace messages appear in the order they are emitted. If you set the TraceMode attribute to the SortByCategory value, then the rows appearing in the Trace Information section are sorted by category name. The category to which each row belongs is determined by the method used to emit the message.

Emitting Trace Messages

An ASP.NET page populates its trace log using methods exposed by the TraceContext class. An instance of this class is created when the HTTP request is set up for execution. The tracer object is then exposed through the Trace property of the HttpContext class and is also mirrored by the Trace property on the Page class.

The TraceContext class has quite a simple interface, featuring a couple of properties and just as many methods. The properties are IsEnabled and TraceMode. IsEnabled is a read-only Boolean property that indicates whether tracing is enabled. The value that this property returns is affected by the trace attribute on the @Page directive as well as the enabled attribute in the trace section of the web.config file. The TraceMode property gets and sets the order in which the traced rows will be displayed on the page. The property is of type TraceMode—an enumeration that includes special values such as SortByCategory and SortByTime.

To emit messages, you can use either the Write or Warn methods. Both methods have three overloads which behave in the same way. Write and Warn are nearly identical methods. The only visible difference is that Warn always outputs messages in red.

The Write method has three overloads:

Public Sub Write(msg As String)
Public Sub Write(cat As String, msg As String)
Public Sub Write(cat As String, msg As String,
    e As Exception);
The simplest overload just emits the specified text in the message column. In the second overload, the first string argument represents the name of the category you want to use for the message, which is the second argument. The category name can be used to sort trace information and is any name that makes sense to the application to better explain the message. Finally, the third overload adds an extra Exception object in case the message is tracing an error. In this case, the text in the message column is created by the concatenation of the text you specify as an argument and the message about the exception.

Although the text output by both Write and Warn is rendered as HTML pages, no special styling or formatting is applied. The strings are written as plain text, so if you attempt to use any special formatting such as bold tags, all you'll get is a trace message with visible <b> and </b> substrings attached.

The ASP.NET Trace object is accessible without a fully qualified name from the source code of the .aspx page or from the codebehind class. Custom controls embedded in the page and their codebehind classes can also access the tracing subsystem directly. Other classes can't, however. Suppose that your codebehind class delegates an external class to accomplish some tasks. How could this worker class perform tracing in the ASP.NET page? In the context of the worker class, the page Trace object is not available, at least not in its unqualified, direct form. External classes that want to emit text in the trace log of the current HTTP request must call the tracer using the following expression:

System.Web.HttpContext.Current.Trace.Write(cat, msg)

Note that the ASP.NET tracing system does not support its own set of listeners, nor does it support those registered through the diagnostic trace section. In addition, there's no known way to modify the output of the trace to show only a few of the standard tables of data. If you want to enhance or modify it, you need a completely new approach.

The Trace Viewer

ASP.NET also supports application-level tracing through the trace viewer tool. Once tracing has been enabled for the application, each page request routes all the page-specific trace information to the viewer. You can look at the trace viewer by requesting trace.axd from the root application directory (see Figure 2). As I mentioned earlier, you enable the viewer by placing a trace section in the application's web.config file—that is, the configuration file deployed in the root folder:

<configuration>
  <system.web>
     <trace enabled="true" />
  </system.web>
</configuration>

Since the pageOutput attribute is false by default, only the viewer receives trace information. However, each page can individually override this setting by using the trace attribute on the @Page directive. The trace viewer caches no more than the number of requests specified by the requestLimit attribute (10 by default).

Figure 2 Viewing the Trace Viewer
Figure 2 Viewing the Trace Viewer

In brief, the ASP.NET trace viewer acts as a centralized console and gathers all the trace information generated by the pages in a particular application. Each request, up to the maximum number fixed by requestLimit, is identified by a row in the viewer's interface and can be consulted until the viewer's cache is cleared (see Figure 2).

You activate the viewer by requesting the trace.axd URL on the root of the application. An AXD is a special resource type that is resolved through the aspnet_isapi.dll ISAPI extension, as Figure 3 shows. At first, the page in Figure 2 is displayed. After that, the viewer automatically tracks all requests and caches the full trace for each. When the request limit is reached, no other request will be cached until the log is manually cleared. You can click on each View Details link to see the trace for that particular request.

Figure 3 Requesting a Trace
Figure 3 Requesting a Trace

The built-in mechanism for tracing is perfect for ASP.NET in the sense that it provides a good deal of runtime information and allows you to assert and verify dynamic values of internal structures. On the other hand, it doesn't show all the information you might want to track, and the text-based user interface is boring. The trace information is appended to the page after the last lifecycle event of the page fires. As you know, an ASP.NET page works by handling the events it receives from the HttpRuntime environment. These events describe the page lifecycle and move from Page_Init to Page_Unload through a handful of situations including Page_Load and Page_PreRender. When the final event is being generated, the HTML code for the browser has not been prepared yet. When the page renders, the HTML is temporarily cached in the output stream and left at the disposal of registered modules for further processing. As a result, the code in the page can't even access the HTML information attached as trace output. This closed structure, along with my dislike for the bland text-based user interface, sent me on a quest for an alternative scheme.

Before discussing MyTracer, I should make a point about tracing in general. While you should pay careful attention not to deploy pages with the trace attribute turned on, tracing calls can be left in the code for future use. Both the Write and Warn methods, in fact, return immediately if tracing is not enabled. This may have an impact on performance, but it will be negligible, especially since the tracing mechanism can be turned on declaratively without touching the code. By simply changing the trace statement's enabled attribute in the web.config file, you enable tracing through the viewer. In this way, you can monitor the application after some user feedback, for example, without stopping it. For the viewer to work, though, trace information must be conditionally emitted.

The MyTracer Tool

There are other ways to write an alternative tracer tool in ASP.NET 1.x. You could write it as an HTTP module and use both the HttpApplication events and the ASP.NET object model to collect the runtime information to display. In addition, you could make it expose methods and events to interact with the page. In this column, I'll propose a different approach that exploits the rich user interface of Windows Forms applications and integrates with Visual Studio .NET. At the end of the day, MyTracer is a Windows Forms program that embeds the WebBrowser ActiveX® control to display Web pages and makes use of a Web Service as the bridge between ASP.NET and Windows. Figure 4 shows the tool in action on a sample page.

Figure 4 MyTracer Tool
Figure 4 MyTracer Tool

The tool navigates to the specified page and displays it. If the page has been configured to work with MyTracer, some page-specific information is exported and made available to a helper Web Service located in the same virtual folder as the application. The Windows Forms application handles the DocumentComplete event and, when ready, connects to the Web Service to download the ASP.NET runtime information. After that, it simply populates the various tabs with page-specific data such as the contents of Session, Application, Cache, the view state, the input form, server variables, and even the list of the server controls in the page.

MyTracer contains three components—the Windows Forms program shown in Figure 4, the aforementioned Web Service, and a Web user control that the traced page must contain. The user control—mydebugtool.ascx—hooks up the key events in the page lifecycle and stores information into a DataSet object. When the page unloads, the DataSet is copied into a database in a way that closely resembles the SQL Server™ implementation of the ASP.NET session state. The Web Service retrieves any data from the database and serializes it down to the Windows Forms application. The data always travels as a DataSet with one child table per tab in the user interface. The database and the Web Service each have extremely simple structures. In particular, the DataSet is persisted as an XML DiffGram so that it is treated and moved as a string. Figure 5 depicts the overall architecture.

Figure 5 MyTracer Architecture
Figure 5 MyTracer Architecture

Enabling MyTracer

To be traced, a page must contain the MyDebugTool user control. The control has a simple user interface that tells the user that the tool is in action. (See the link at the top of this article.) The control hooks up the Page_Unload and Page_PreRender events and caches application- and request-specific information such as Application, Cache, and Session. The data container is a DataSet object that is instantiated within the control's constructor and is persisted to a SQL Server database as the last action when the control is signaled that the page is unloading. Given an existing ASP.NET page, to enable MyTracer you must register the MyDebugTool control and add it to the control collection. A user control is registered through the @Register directive:

<%@Register TagPrefix="msdn" TagName="debug" Src="MyDebugTool.ascx"%>

Note that TagPrefix and TagName can take any value and that the user control must reside in the same Web domain as the application. You can place an instance of the control anywhere in the page, even at the very top. Given the registration I just showed, you use the following code:

<msdn:debug runat="server" ID="mytracer" UserKey="dino" />

The ID is not strictly necessary but, as you'll see later, you must assign one if you want to access view state information. The UserKey attribute—a public property on the user control class—is used to identify the database row where all the information will be parked. It has to be a unique name, but there are no other requirements. When you're done with these configuration tasks, point any browser to the page. The page displays a blue strip with some text stating that the page is now under the control of the MyTracer tool. As soon as the page is sent to the browser, the database is updated. The connection string for the database can be set at will using the ConnString property. It defaults to a SQL Server database called MyTracer with a table named InternalCache. The table features two fields—UserKey and Data. UserKey is a string; Data is a Text field that contains the DiffGram representation of the DataSet that contains all the collected information. As of this writing, the code assumes that a record with the specified userkey already exists in the database. In other words, only UPDATE statements are used and INSERT's are never performed. As long as the following query can be successfully understood and executed, you can use any database on any server with any login.

SELECT data FROM internalcache WHERE userkey=@TheUser

The user control creates a number of tables in a temporary DataSet. All tables have two string columns named Key and Value. The control creates tables to hold the contents of the following collections: Request.Headers, Request.Form, Request.Cookies, Request.ServerVariables, and Application, Cache, Session, plus the page's view state and controls. The code in Figure 6 shows how information about the Cache and the page controls is retrieved. In particular, the Cache object contains both user and system information. As the tracer shows, a few slots in the collection are managed by the server-side HTTP runtime and point to internal classes used to contain runtime information. For example, if the session state is configured to be local to the ASP.NET process (which is the default), then the collection of session items is stored as a single object in Cache. The internal session collection is called SessionStateItem, as Figure 7 illustrates.

Figure 7 System Items Stored in Cache (Close Figure 7)

Figure 6 Retrieving Cache and Page Control Information (Close Figure 6)
// *****************************************************************
// Load information from cache
private void LoadFromCache(DataSet info)
{
    DataTable dtCache = CreateKeyValueDataTable("Cache");
    foreach(DictionaryEntry elem in Page.Cache)
    {
        if (ShowAll)
        {
            AddKeyValueItemToTable(dtCache, elem.Key.ToString(),
                DisplayFormat(elem.Value));
        }
        else
        {
            string s = elem.Key.ToString();
            if (!s.StartsWith("ISAPIWorkerRequest") &&
                !s.StartsWith("System"))
                AddKeyValueItemToTable(dtCache, elem.Key.ToString(),
                DisplayFormat(elem.Value));
        }
    }

    dtCache.AcceptChanges();
    info.Tables.Add(dtCache);
    return;
}
// *****************************************************************

// *****************************************************************
// Load information from page's controls
private void LoadFromPageControls(DataSet info)
{
    HtmlForm theForm = null;

    DataTable dtControls = CreateKeyValueDataTable("Controls");
    for (int i=0; i<Page.Controls.Count; i++)
    {
        if (Page.Controls[i] is HtmlForm)
            theForm = (HtmlForm) Page.Controls[i];
        AddKeyValueItemToTable(dtControls, Page.Controls[i].ToString(),
            Page.Controls[i].ClientID);
    }
    dtControls.AcceptChanges();
    info.Tables.Add(dtControls);

    if (theForm == null)
        return;

    DataTable dtFormControls = CreateKeyValueDataTable("FormControls");
    for (int i=0; i<theForm.Controls.Count; i++)
        AddKeyValueItemToTable(dtFormControls,
            theForm.Controls[i].ToString(),
            theForm.Controls[i].ClientID);
    dtFormControls.AcceptChanges();
    info.Tables.Add(dtFormControls);

    return;
}
// *****************************************************************

Looking at Figure 7, you can see that all the system items stored in the cache have a name that starts with either "System" or "ISAPIWorkerRequest." If you want to see the items that any sessions in the application placed in the cache, set the ShowAll property of the MyDebugTool control to false. In Figure 7, some values in the Value column are wrapped by curly brackets, meaning that the item contains an object and the displayed string is the name of the class. The following code examines the object's type and returns the class name or the actual value, as appropriate:

string DisplayFormat(object o)
{
    return (o is string || o.GetType().IsPrimitive
       ?Convert.ToString(o)
       :"{ " + o.ToString() + " }");
}
The IsPrimitive property on the Type class returns true if the type is a numeric type, char, or Boolean. Note that in this context strings are not considered primitive types; rather, they're treated as a reference type and as instances of the System.String class.

The code used to extract all the information stored in the Cache is similar to the code used to extract the contents of Application, Session, or any Request collection.

To list the controls in the page, you loop through the Page.Controls collection. Note, though, that the collection covers only the first level of controls—that is, only the controls that are direct children of the body element in the overall HTML model of the page. In practice, if you were to loop through the Page.Controls collection, you get a very small number of controls—mostly literals and the ASP.NET unique form control. By design, the server controls that make the page work are contained within a server-side form. To access all of them, you must perform a second loop based on the form's Controls collection. To obtain the full control tree, as in the default ASP.NET trace mode, you must recursively loop through all the Controls collections of all listed controls. The MyTracer tool displays the ID of each control. Note that, by design, all server controls must have a client ID. This ID has to be unique and is normally assigned based on the value of the server-side ID property. If the server control doesn't have an ID, the ASP.NET runtime generates one that is guaranteed to be unique within the page. To access the client-side ID, use the ClientID property.

The Helper Web Service

Architecturally speaking, a Web Service is rather interesting because it can be used to create a bridge between ASP.NET and desktop applications and even between different ASP.NET applications. In this case, the Web Service is the element that glues together the Windows Forms program and the monitored ASP.NET page. It provides a common interface to allow the application to obtain ASP.NET-specific information. I used a Web Service here because it was easy to work with. Other possibilities exist, too. For example, you could use an external Windows-based service that operates through remoting, getting data from the user control and handing it over to the Windows Forms application. Similarly, the choice to use a database is a personal one. You could use global memory within the service instead. Overall, the StateServer and SqlServer options that you have for storing the session state of an ASP.NET app can be replicated for MyTracer.

The programming interface of the Web Service is extremely simple (see Figure 8). The GetInfo method executes a query against the database and gets back a string that represents the XML DiffGram representation of the original DataSet. GetInfo deserializes the string and rebuilds a valid DataSet object using the DataSet's ReadXml method. Next, the newly created and populated DataSet object is returned to the caller. Note that if the UserKey is left unchecked, the Web Service is susceptible to a SQL injection exploit. Ideally you would want to either scrub the UserKey field or use a parameterized query.

Figure 8 Web Service Programming Interface (Close Figure 8)
<%@ WebService language="C#" class="MyDebugTool" %>

using System;
using System.Web.Services;
using System.Data;
using System.IO;
using System.Data.SqlClient;

[WebService(Namespace="MsdnMag.CuttingEdge")]
public class MyDebugTool
{
    [WebMethod]
    public DataSet GetInfo(string connString, string userKey)
    {
        SqlDataAdapter adapter = new SqlDataAdapter(
            "SELECT data FROM internalcache WHERE userkey='" +
            userKey + "'", connString);
        DataTable tmp = new DataTable();
        adapter.Fill(tmp);

        DataSet ds = new DataSet();
        StringReader reader = new
            StringReader(tmp.Rows[0]["data"].ToString());
        ds.ReadXml(reader);
        ds.AcceptChanges();
        return ds;
    }
}

The Windows-based Application

The MyTracer application employs a tabstrip control and the WebBrowser ActiveX control to display the page, thus triggering the user control. When the page download is complete, the application calls the Web Service and the GetInfo method to bind data to the other tab pages. The following code shows how to write handlers for a couple of WebBrowser's events—StatusTextChange and DocumentComplete:

// Hook up the StatusTextChange event
DWebBrowserEvents2_StatusTextChangeEventHandler e1;
e1 = new
  DWebBrowserEvents2_StatusTextChangeEventHandler(StatusTextChange);
webBrowser.StatusTextChange += e1;

// Hook up the DocumentComplete event
DWebBrowserEvents2_DocumentCompleteEventHandler e2;
e2_= new
  DWebBrowserEvents2_DocumentCompleteEventHandler(DocumentComplete);
webBrowser.DocumentComplete += e2;
The information available for the event is packed in a custom event data class whose name matches that of the event handler. For example, the name of the class is DWebBrowserEvents2_DocumentCompleteEvent for the DocumentComplete event.

All the page tabs have a common structure, containing one or more DataGrid controls that are bound to one of the tables in the DataSet retrieved via the Web Service.

The tracer application also retrieves the page's view state (see Figure 9). This feature goes beyond the capabilities of the ASP.NET default tracer, which is limited to displaying the size (in bytes) of the view state for each control on the page. MyTracer only provides information about the view state elements stored within the page's view state. It doesn't cover the state elements created by individual controls. As I discussed in the February 2003 installment, the ASP.NET view state is created by aggregating the view state of each constituent control with that of the host page. The view state of a page or a control is a StateBag object exposed by the ViewState property. The rub, though, is that ViewState is a protected property and this level of protection makes the information inaccessible from within external components.

Figure 9 View State
Figure 9 View State

So how can the user control retrieve the view state of the host page? One possibility that I haven't yet fully explored and investigated is reflection. In theory, the Reflection API in the common language runtime allows you to programmatically access internal and non-public elements of a class. However, it has yet to be proven that this can be done from within the security context of an ASP.NET application. (By the way, if you experiment and obtain some interesting results, please let me know!)

MyTracer uses a more direct but less powerful way to read the page's view state. The user control simply makes it possible for the page (and subsequently for the page's author) to deliberately expose its view state to the tool. The MyDebugTool control features a BindViewState method that the monitored page can call at any time and pass its view state object, as shown here:

void Page_Load(object sender, EventArgs e) {
   •••

   // Bind the viewstate to display
   mytracer.BindViewState(ViewState);
}
In this way, the page can also pass on any view state object it may hold—for example, the view state that a custom control may decide to expose for debugging purposes at development time. A simple trick you can use to look into a control's view state is to create a wrapper class for the control that exposes only the ViewState property. For example:
public class MyDataGrid : DataGrid
{
    public StateBag ExternalViewState
    {
        get {return ViewState;}
    }
}

Next, use the new control in the page and bind the exposed copy of the view state to the MyTracer application. The trick doesn't work if the control class is sealed and can't be derived any further. So if you're developing a commercial control and don't want users to spy on your control's state, sealing the class for inheritance would do the trick. However, this choice has its pros and cons. For example, I wouldn't buy a third-party control that I couldn't inherit from. Also, note that I'm simply talking about spying the view state, not modifying it. Modifying the view state is virtually impossible, as I discussed in my February 2003 column.

Putting It All Together

The MyTracer application is useful when you're developing and testing a particular ASP.NET page. The tool runs the page, captures any runtime information, and displays it to you in a friendly way. It also provides more information than you can view while tracing the page through traditional methods. This version of the code doesn't support the output of custom messages, nor does it capture the output you sent using the Trace object. However, adding a message container is as easy as adding a new Write-like method to the user control and collect all passed strings in another table.

MyTracer's purpose is to serve as a helper tool that can replace the browser while testing an ASP.NET application, even within Visual Studio .NET. In the Configuration Properties of your Visual Studio ASP.NET project, check the Start external program option, type in the path to MyTracer, and set the page URL as the command-line argument for the tool (see Figure 10).

Figure 10 Setting Up MyTracer in Visual Studio .NET
Figure 10 Setting Up MyTracer in Visual Studio .NET

You should note that while using MyTracer, the ASP.NET debugging feature doesn't work very well. If you need to step into the code, nothing works better than the debugger, where the Quick Watch feature allows you track just about any value in the application. If you don't need to debug, then MyTracer is certainly more helpful than the browser. To enable the debugger again, click on the Start project option in the same dialog (see Figure 10).

In the code download for this column, you'll find the source code for MyTracer as well as a sample ASP.NET project to experiment with. I can't wait to hear your comments!

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Espositois an instructor and consultant based in Rome, Italy. Author of Building Web Solutions with ASP.NET and ADO.NET and Applied XML Programming for .NET, both from Microsoft Press, he spends most of his time teaching classes on ASP.NET and speaking at conferences. Dino is currently writing Programming ASP.NET for Microsoft Press. Get in touch with Dino at dinoe@wintellect.com.


The XML Files: Web Services and DataSets

XML Files
Web Services and DataSets
Aaron Skonnard

Code download available at: XMLFiles0304.exe (279KB)
Contents
Levels of Interoperability
The DataSet Problem
Toolkit Bindings
Avoiding DataSet
Typed DataSets
Customizing the Schema
The Ultimate Solution
A .NET Framework Workaround
Wrapping It Up

Programmers using Visual Basic® 6.0 have long bowed to the altar of the ADO recordset. It's probably the most commonly used data structure in Windows®-based applications today. The ADO.NET DataSet is poised to play a similar role in the realm of managed Windows-based applications.

When you need to extract information from a database, present it to the user, and apply updates, the DataSet is often the best fit. The DataSet has been used quite successfully for a variety of solutions targeting the Microsoft® .NET Framework. In the case of Web Services, the DataSet can cause interoperability issues.

If you use dynamic data structures like the DataSet it can be harder for developers who use other toolkits to consume your Web Service. Using typed DataSets in combination with slightly customized Web Services Description Language (WSDL) definitions, however, gets around this issue. Developers need to understand the issues surrounding all dynamic data structures (like the DataSet) before using them in Web Service interfaces.

Levels of Interoperability

The ability to process data regardless of where it comes from is the most primitive level of interoperability. I like to call this "data interoperability." XML 1.0 was specifically developed to facilitate data interoperability.

Data serialized into XML 1.0 is easy to extract and manipulate on any platform using any programming language for which an XML processor exists, which encompasses practically all languages today. Achieving data interoperability still requires you to write code against an XML API exposed by your XML processor, but you don't have to deal with the byte stream directly (see Figure 1).

Figure 1 Data Interoperability Through XML
Figure 1 Data Interoperability Through XML

Applying XML to distributed application technology (Web Services) simplifies interoperability between heterogeneous distributed systems. Web Services can be built using XML APIs directly. This approach gives you full control over message processing, but it requires you to implement a great deal of common Web Service infrastructure, which can decrease productivity and introduce bugs that can lead to more interoperability problems.

Distributed technology vendors (like Microsoft) have been busy developing Web Service toolkits that provide this common infrastructure, helping reduce the XML API code required of the Web Service developer. The Microsoft ASP.NET WebMethod framework is one such toolkit that removes almost all traces of the lower-level System.Xml APIs. Web Service toolkits accomplish this by defining mappings between application types and XML Schema types, thereby automating the translation between XML documents and object instances at run time.

For example, consider the following WebMethod named CalculateMortgage that takes a MortgageInfo object as input and returns a MortgagePayments object to the caller:

[WebService(Namespace="http://example.org/mortgage")]
public class MortgageService : WebService
{
   [WebMethod]
   public MortgagePayments CalculateMortgage(
      MortgageInfo minfo)
   {
      ... // calculate mortgage
      return new MortgagePayments();
   }
}
Within this method, the developer is simply working with objects, but the [WebMethod] attribute tells the ASP.NET infrastructure to treat this method as a Web Service operation. So in this case the infrastructure automatically maps the MortgageInfo and MortagePayment classes to XML Schema type definitions.

Figure 2 shows the MortgageInfo and MortgagePayments C# class definitions used by the CalculateMortgage WebMethod, while Figure 3 shows the XML Schema type definitions generated by the ASP.NET infrastructure. With these XML schema definitions in place, it's clear what the other side should expect in terms of XML.

Figure 3 XML Schema Type Definitions (Close Figure 3)
•••
<s:complexType name="MortgageInfo">
   <s:sequence>
     <s:element name="amount" type="s:double" />
     <s:element name="years" type="s:double" />
     <s:element name="interest" type="s:double" />
     <s:element name="annualTax" type="s:double" />
     <s:element name="annualInsurance" type="s:double" />
   </s:sequence>
</s:complexType>

<s:complexType name="MortgagePayments">
   <s:sequence>
     <s:element name="MonthlyPI" type="s:double" />
     <s:element name="MonthlyTax" type="s:double" />
     <s:element name="MonthlyInsurance" type="s:double" />
     <s:element name="MonthlyTotal" type="s:double" />
   </s:sequence>
</s:complexType>
•••
Figure 2 Class Definitions (Close Figure 2)
public class MortgageInfo
{
   public double amount;
   public double years;
   public double interest;
   public double annualTax;
   public double annualInsurance;
}

public class MortgagePayments
{
   public double MonthlyPI;
   public double MonthlyTax;
   public double MonthlyInsurance;
   public double MonthlyTotal;
}

Web Service toolkits make it easy to ignore XML since the Web Service developer deals only with objects. The toolkit on the other end of the wire, however, may also want to map the XML to objects. For example, a developer who uses the Java language might use something like Apache's Axis toolkit to consume this Web Service. To do so, he would use the WSDL2Java utility to automatically generate Java classes from the XML Schema type definitions (see Figure 4). This makes it possible for the Java developer to also work with objects without having to drop down to the lower-level Java XML APIs.

Figure 4 Apache Axis-generated Java Class Definitions (Close Figure 4)
/**
 * MortgageInfo.java
 *
 * This file was auto-generated from WSDL
 * by the Apache Axis WSDL2Java emitter.
 */

public class MortgageInfo  implements java.io.Serializable {
    private double amount;
    private double years;
    private double interest;
    private double annualTax;
    private double annualInsurance;

    public MortgageInfo() {
    }

    public double getAmount() {
        return amount;
    }

    public void setAmount(double amount) {
        this.amount = amount;
    }
    ... // remainder ommited for brevity
}

/**
 * MortgagePayments.java
 *
 * This file was auto-generated from WSDL
 * by the Apache Axis WSDL2Java emitter.
 */

package mortgage_tools;

public class MortgagePayments  implements
  java.io.Serializable {
    private double monthlyPI;
    private double monthlyTax;
    private double monthlyInsurance;
    private double monthlyTotal;

    public MortgagePayments() {
    }

    public double getMonthlyPI() {
        return monthlyPI;
    }

    public void setMonthlyPI(double monthlyPI) {
        this.monthlyPI = monthlyPI;
    }
    ... // remainder omitted for brevity
}

Now let's discuss the next level of interoperability—what I like to call "toolkit interoperability." Toolkit interoperability means that I can write Web Service code in one toolkit and easily consume it using another toolkit without dropping down to the lower-level XML APIs. There must be an equivalent object mapping or the other side will have to drop down and deal with XML directly (this is illustrated in Figure 5).

Figure 5 Toolkit Interoperability
Figure 5 Toolkit Interoperability

Toolkit interoperability is harder to achieve due to the many differences in programming languages and type systems. The DataSet is one type that presents some interesting challenges to toolkit interoperability.

The DataSet Problem

Most Web Service toolkits do a good job mapping simple class definitions like the ones shown earlier. They begin to experience problems, however, when derivation and substitution come into play or when dynamic types are used, such as collections, hash tables, or DataSets. You should consider the following .NET Framework signature:

[WebMethod]
public DataSet GetAuthors()
{
    ... // return DataSet filled with authors
}
What should DataSet be mapped to in terms of XML Schema? The DataSet is a polymorphic type whose actual layout isn't determined until run time, after the DataSet has been filled with data.

The following XML Schema definition contains the element declaration for GetAuthorsResponse. Notice that GetAuthorsResponse contains an optional element named GetAuthorsResult, which contains two children: an XMLSchema schema element (s:schema), followed by a wildcard (s:any) indicating that any element from any namespace can be provided:

<s:element name="GetAuthorsResponse">
   <s:complexType>
      <s:sequence>
         <s:element minOccurs="0" maxOccurs="1"
            name="GetAuthorsResult">
            <s:complexType>
               <s:sequence>
                  <s:element ref="s:schema" />
                  <s:any />
•••
</s:element>
This indicates that a schema will be provided at run time that will describe the XML that follows it. Here's what this would look like in the SOAP response:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:s="http://www.w3.org/2001/XMLSchema"
>
  <soap:Body>
    <GetAuthorsResponse
      xmlns="http://example.org/dataset-service">
      <GetAuthorsResult>
        <s:schema> *** schema goes here *** </s:schema>
        *** xml goes here ***
•••
</soap:Envelope>

Since the toolkit won't be able to determine the actual types used in the DataSet until run time, the next question is what the XML Schema definition (just shown) should map to in other environments like Apache's Axis toolkit.

Toolkit Bindings

At design time, when the Java developer runs WSDL2Java, there isn't enough information in the schema definition to do anything other than map the s:any element to java.lang.Object, as follows:

•••
public class GetAuthorsResult  implements java.io.Serializable {
    private java.lang.Object any;

    ... // omitted for brevity
}

As it turns out, Axis can't even get that far because it doesn't like the reference to the s:schema element that precedes s:any; it throws an exception and fails to produce anything in this case. If the schema reference didn't exist, however, the Java class would look something like the one shown previously.

Since neither the developer nor the toolkit have any additional type information (about the structure of the author information in this case), the only thing that makes sense is for the toolkit to provide a DOM tree in the field containing the parsed XML. Hence, in order for Java developers to consume this Web Service, they'll have to drop down and use their equivalent DOM API directly. The code in Figure 6 shows you how to do this.

Figure 6 Axis Client Program (Using XML APIs) (Close Figure 6)
import java.io.*;
import org.example.*;
import org.w3c.dom.*;
import org.apache.xerces.dom.*;
import org.apache.axis.message.*;

public class AuthorsClient {
   static void main(String[] args) throws Exception
   {
       DataSetServiceSoapStub stub =
           new DataSetServiceSoapStub(new java.net.URL(
           "http://localhost/ds/ds.asmx"), null);

       GetAuthorsAsTypedDataSetResult result =
         stub.getAuthors();
       Object any = result.getAny();
       // it's really a DOM tree
       Element docElement = (Element)any;

       NodeList authors = docElement.getElementsByTagNameNS(
           "http://example.org/dataset", "authors");
       for (int i=0; i<authors.getLength(); i++)
       {
           Element authorsElem = (Element)authors.item(i);
           ElementImpl fnameElem =
              (ElementImpl)authorsElem.getElementsByTagNameNS(
              "http://example.org/dataset","au_fname").item(0);
           System.out.println(fnameElem.getTextContent());
       }
   }
}

You will experience this issue with any toolkit that doesn't have a special case for DataSets in the .NET Framework. You'll even have to deal with the DOM API directly using the Microsoft SOAP Toolkit for COM developers.

According to the definitions I provided earlier there is still data interoperability in this situation (it's just XML), but a level of toolkit interoperability is lost since the consuming developer is now required to deal with the raw XML. If I know I'm returning a collection of author records with a well-defined format, there are a few ways I could make it easier for users of other toolkits to process my data without touching the XML APIs directly.

Avoiding DataSet

Probably the easiest way around this is to not use DataSets in your Web Service interfaces. For example, in this case you could define a class that models author information and declare the GetAuthors method as follows:

public class Author
{
    public string id;
    public string au_fname;
    public string au_lname;
    •••
}

[WebMethod]
public Author[] GetAuthors()
{
    DataSet authorsDS = GetAuthorsFromDB();
    ... // walk through DataSet and generate
        // Author array to return
}

Here I'm manually mapping a DataSet to the static type Author (since I'm probably only using the DataSet here to retrieve the data from the database, it probably makes more sense to use IDataReader directly since it would provide a more efficient solution). In this case there would be a natural mapping between the Author class and the equivalent XML Schema definition, which also map nicely back to an equivalent Java class (like the CalculateMortgage example shown earlier).

With something like this in place, the Axis developer would be able to write code without touching the XML APIs directly (see Figure 7). Another way to accomplish this would be to generate a typed DataSet from an XML Schema definition and use it instead of the generic DataSet.

Figure 7 Axis Client (No XML APIs) (Close Figure 7)
import java.io.*;
import org.example.*;

public class AuthorsClient {
   static void main(String[] args) throws Exception
   {
      DataSetServiceSoapStub stub =
          new DataSetServiceSoapStub(new
          java.net.URL("http://localhost/ds/ds.asmx"), null);
      Author[] authors = stub.getAuthors();
      for (int i=0; i<authors.length; i++)
         System.out.println(authors[i].getAu_Fname());
   }
}

Typed DataSets

Typed DataSets are classes that derive from DataSets and expose strongly typed members that depict a specific view of the data. Unlike generic DataSets, typed DataSets are bound to a specific structure, an XML Schema definition, at design time. You can automatically generate typed DataSets from XML Schema definitions using xsd.exe or the built-in designers in Visual Studio® .NET.

Figure 8 contains an XML Schema definition that describes the author records I intend to return from my Web Service. After running this schema through xsd.exe using the /dataset switch (or using the Visual Studio .NET designer), I'll have a new class named AuthorSet that derives from DataSet, as shown here:

[Serializable()]
public class AuthorSet : DataSet {

    private authorsDataTable tableauthors;

    ... // strongly typed accessors
}

Figure 8 Typed DataSet XML Schema Definition (AuthorSet.xsd) (Close Figure 8)
<xs:schema id="AuthorSet"
  targetNamespace="http://example.org/dataset"
  xmlns="http://example.org/dataset"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  elementFormDefault="qualified"
  ...
>
  <xs:element name="AuthorSet" ...>
    <xs:complexType>
      <xs:choice maxOccurs="unbounded">
        <xs:element name="authors">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="au_id" type="xs:string" />
              <xs:element name="au_lname" type="xs:string" />
              <xs:element name="au_fname" type="xs:string" />
              ...
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:choice>
    </xs:complexType>
    ...
  </xs:element>
</xs:schema>

The infrastructure knows what the schema definition should be for this typed DataSet, so it shouldn't have to provide a schema at run time anymore (although it still does). Now I can create a new WebMethod that returns an AuthorSet object:

[WebMethod]
public AuthorSet GetAuthorsAsTypedDataSet()
{
    AuthorSet aus = new AuthorSet();
    sqlDataAdapter1.Fill(aus);
    return aus;
}

Now the infrastructure has enough information to inform clients at design time about the structure of the author records in the returned set. When the ASP.NET infrastructure generates the WSDL and XML Schema definitions, however, it imports the typed DataSet's XML Schema definition and uses a constrained wildcard, as shown in Figure 9.

Figure 9 GetAuthorsAsTypedDataSet XML Schema Definition (Close Figure 9)
<definitions
  xmlns:s="http://www.w3.org/2001/XMLSchema"
  targetNamespace="http://example.org/dataset-service"
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  ...
>
  <import namespace="http://example.org/dataset"
   location="http://localhost/ds/ds.asmx?schema=AuthorSet" />
  <types>
    <s:schema
     elementFormDefault="qualified"
     targetNamespace="http://example.org/dataset-service"
    >
      <s:import namespace="http://example.org/dataset" />
      ...
      <s:element name="GetAuthorsAsTypedDataSetResponse">
        <s:complexType>
          <s:sequence>
            <s:element minOccurs="0" maxOccurs="1"
             name="GetAuthorsAsTypedDataSetResult">
              <s:complexType>
                <s:sequence>
                  <s:any
                   namespace="http://example.org/dataset" />
                </s:sequence>
              </s:complexType>
            </s:element>
          </s:sequence>
        </s:complexType>
      </s:element>
      ...
    </s:schema>
    ...
</definitions>

Since there is no longer a reference to the s:schema element, WSDL2Java is able to generate Java classes successfully. And since the import statements give it access to the XML Schema definition for AuthorSet, it can generate equivalent Java classes that represent the author information. Axis, however, still maps the wildcard (s:any) to java.lang.Object, which will contain a DOM tree at run time. This gets me further than before, but still leaves the client working with the DOM API directly.

Customizing the Schema

Since you know the wildcard slot is always going to contain an AuthorSet, you might be tempted to just modify the schema definition to reference the AuthorSet element instead of the wildcard. The wildcard, however, is actually a placeholder for the generic DataSet schema (which doesn't make much sense since the wildcard is constrained to the http://example.org/dataset namespace). In other words, when you execute the WebMethod, the GetAuthorsAsTypeDataSetResult element will contain a schema element followed by a diffgram element, and the diffgram element will contain the AuthorSet element.

But since you know the structure ahead of time, you don't need to provide a run-time schema or a diffgram element; you can simply return an AuthorSet element. The way to do this is to define a new WebMethod that doesn't return a DataSet, but rather an XmlNode. Then if you return an XmlDataDocument that wraps the typed DataSet object, only the AuthorSet element is returned to the client, without the run-time schema or diffgram elements:

[WebMethod]
public XmlNode GetAuthorsAsXml()
{
    AuthorSet aus = new AuthorSet();
    sqlDataAdapter1.Fill(aus);
    return new XmlDataDocument(aus);
}
XmlDataDocument is inefficient since it roughly doubles the size of the original DataSet. Since it's transient, it may be acceptable.

Specifying that the method returns an XmlNode will also result in a wildcard in the generated schema, but now you can replace the wildcard with a reference to the AuthorSet element, as shown here:

•••
<s:element name="GetAuthorsAsXmlResponse">
  <s:complexType>
    <s:sequence>
      <s:element minOccurs="0" maxOccurs="1"
        name="GetAuthorsAsXmlResult">
        <s:complexType>
          <s:sequence>
            <s:element ref="x:AuthorSet" />
          </s:sequence>
        </s:complexType>
•••

You'll have to make sure clients no longer retrieve the auto-generated WSDL file once you start down this path. Instead you'll want to specify a custom WSDL file location by using the WebServiceBindingAttribute (see the Location property) or by providing a static WSDL file on your server and disabling the automatic WSDL generation. You can disable the automatic WSDL generation by adding the following section to your web.config file:

<webServices>
  <protocols>
    <remove name="Documentation" />
  </protocols>
</webServices>
Now the Web Service can still work with DataSets and other clients can process the results as an array of objects. Figure 10 shows what an Axis client would look like in this case.

Figure 10 Axis Client from Typed DataSet (Close Figure 10)
import org.example.*;

public class AuthorsClient {
   static void main(String[] args) throws Exception
   {
      DataSetServiceSoapStub stub =
        new DataSetServiceSoapStub(new java.net.URL
        ("http://localhost/ds/ds.asmx"), null);
      GetAuthorsAsTypedDataSetResult res =
        stub.getAuthorsAsTypedDataSet();
      AuthorSet aset = res.getAuthorSet();
      Author[] authors = aset.getAuthors();
      for (int i=0; i<authors.length; i++)
         System.out.println(authors[i].getAu_Fname());
   }
}

If this seems like a big hassle just to use DataSets, you can always avoid them in your Web Service interfaces or simply require your clients to deal with the XML directly. It all comes down to what you want other toolkits to see.

The Ultimate Solution

This problem could eventually go away if the industry defined a standard XML Schema definition for representing database resultsets and all toolkits properly supported it. Then toolkits could map their native resultset types to the standardized schema. The same holds true for other dynamic types like collections, hash tables, lists, queues, and so on. Standardizing XML Schema definitions for such types would have the effect of promoting toolkit interoperability across a wider range of extremely useful and commonly used data structures.

If this were a reality, you could return a generic DataSet (like the first GetAuthors example) and the Axis toolkit could map it to a JDBC WebRowSet (or something equivalent). Then, the Java developer could bind the WebRowSet to a UI component, allow the user to modify the data, and pass an updated set back to the server—all with very minimal code. The updated set could then be deserialized back into a DataSet on the server in order to apply the changes to the data.

This solution makes the most sense if you think about why you would want to use a DataSet in a Web Service. DataSets are typically used when you need to present data to the user in a generic fashion (usually in a grid), allow the user to make changes, and apply updates back to the database. The techniques described here don't support this pattern (since you don't get data binding or diffgram support with DOM trees or arrays), so it's a lot like pounding a square peg into a round hole.

Of course, standardizing anything in this industry is much easier said than done. Don't hold your breath waiting for such standards to become codified and supported throughout today's toolkits.

A .NET Framework Workaround

You've probably seen trade show demos in the past where the presenter returned a DataSet to a .NET Framework client and somehow it magically became a DataSet on the other end. This works because the Web Service infrastructure in the .NET Framework looks for a special flag in the runtime-provided XML Schema definition called IsDataSet. For example, in the case of the GetAuthorsAsTypedDataSet example, the AuthorSet element declaration will look like this in the runtime-provided schema:

•••
<xs:element name="AuthorSet" msdata:IsDataSet="true">
•••

The IsDataSet flag is the "wink" to the client, telling it to deserialize the XML back into a DataSet. Obviously, this only works for .NET Framework clients or other toolkits that have provided this hardcoded special case (there are none that I know of today). The code in Figure 11 shows how a Visual Basic® .NET client could invoke both GetAuthors and GetAuthorsAsTypedDataSet and easily map the returned DataSet to a DataGrid (on a Windows Form).

Figure 11 Mapping DataSet to a DataGrid (Close Figure 11)
Private Sub Button1_Click(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles Button1.Click
    Dim svc As DataSetService = New DataSetService()
    Dim ds As DataSet = svc.GetAuthors()
    DataGrid1.DataSource = ds.Tables.Item(0)
End Sub

Private Sub Button2_Click(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles Button2.Click
    Dim svc As DataSetService = New DataSetService()
    Dim auset As AuthorSet = svc.GetAuthorsAsTypedDataSet()
    DataGrid1.DataSource = auset.authors
End Sub

It would be just as easy across all toolkits and a variety of database APIs if there was a standardized schema for database resultsets that everyone supported. Ironically, once you have something like this in place, it's actually more tedious to work with an array of Author objects when your goal is to present the data to the user for generic manipulation.

Wrapping It Up

XML and Web Services provide basic data interoperability today. If you're willing to work with XML APIs, there isn't a Web Service in the world that you can't consume. If, however, you'd rather tuck the XML APIs under the rug and never deal with them again, be careful with dynamic types like the DataSet. Using the generic DataSet in Web Service interfaces will force developers using other toolkits to deal with the XML directly. Using typed DataSets in combination with slightly customized WSDL definitions gets around this issue by exposing the results as a simple array of objects. The ultimate solution, however, would be to standardize an XML Schema definition for representing database resultsets that could be supported across all toolkits.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnardis an instructor/researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


Advanced Basics: Visual Studio .NET 2003 Usage Tips

Advanced Basics
Visual Studio .NET 2003 Usage Tips
Ken Spencer

Q When I'm working on a solution with many projects and assemblies, the Visual Studio® .NET IDE seems slower; the same is true when there are fewer assemblies and more code in each assembly. Ctrl+Tab switching between code view and design view takes a lot more time and sometimes the IDE locks up because background compilation hangs. Do you have any suggestions that would help me to improve performance?

A The most likely cause of your slowdown is the background compile process used by Visual Basic® .NET. Background compiling has been around a long time and provides some of the immediate feedback developers find so convenient. For instance, if the compilation finds an error, you can fix it in the editor and the background compiler checks it for you when you move on to another line. Overall, this makes for a productive development process, unless the compiler takes up so many cycles that it slows you down. Of course, if you have a solution that contains many thousands of lines of code, this process may bog down the system.

I don't run into this problem very frequently because I always break down projects into smaller chunks. Then I create a solution that is relevant to the current part of the application I'm working on but also has references to the DLLs in other parts.

That said, Visual Studio .NET 2003 fixes the background compile problem for large projects. Along with this and other improvements, the Visual Studio .NET 2003 IDE seems snappier in general than Visual Studio .NET 2002. These changes make the IDE much better for large projects. My testing was done with the final beta, so you may even see better performance when the final release ships.

There are other features that help performance in large solutions. Often, you may have a solution that groups lots of projects together for convenience, thus allowing you to access each project directly as needed. However, it adds to the build time when you press F5 to run and test your application, and every project in the assembly is built. Now you can configure Visual Studio .NET 2003 to build only your startup projects and any dependencies for those projects. For instance, if you have 20 projects in a solution and only 5 of them are required to be rebuilt each time you run the application, that takes 75 percent of the projects out of the build process during a test run. This should result in much faster test runs, depending, of course, on the total number of projects that you are running and their size.

This feature is controlled by a new setting in the Tools | Options | Environment folder on the Projects and Solutions dialog. The "Only build startup projects and dependencies on Run" option defaults to False. If you select either the Visual C++® or Visual C#® developer, the option is set to True. Figure 1 shows this dialog with the setting changed to True.

Figure 1 New Options in Visual Studio .NET 2003
Figure 1 New Options in Visual Studio .NET 2003

There is another change that may affect large applications during the build process. In the past, you could only set one Visual Basic project as the startup project. This has been true for all versions of Visual Basic, including Visual Studio .NET. Visual Studio .NET 2003, however, allows you to select multiple startup projects, as shown in Figure 2.

Figure 2 Selecting Startup Projects
Figure 2 Selecting Startup Projects

An example of multiple startup projects involves .NET Framework Remoting. If you build a .NET Framework Remoting application and create your own host app, each time you test the application you must make sure the host is running before the client tries to access it. Now you can flag both projects as startup projects in a single solution and each time you press F5, both projects start. This makes solutions really good containers—not only for development but also for testing when you need to open projects and run multiple applications. Stopping any of the startup applications seems to stop the run session and all applications.

Now, consider both the ability to only build startup projects and their dependencies and the ability to have multiple startup projects. Consider an example where your solution has 10 projects. Let's say you have one project that you want to start all the time and there are three other projects it depends on. That only requires the building of four projects. But you can configure another startup project and allow it to have all of the other projects as dependencies. Then you can switch between the two startup projects and change what gets compiled. This makes moving back and forth between build processes as simple as changing the startup projects.

Q I have two questions related to building Windows®-based applications that would both save coding time and possibly improve the performance of my application. First, how can I determine if a DataReader has rows without having to read the data? Second, is there an easier way using Windows Forms to allow a user to select a folder that requires less code?

A One of the changes in Visual Studio .NET 2003 is the DataReader. The new HasRows property enables you to check for rows without reading any data from the DataReader, resulting in faster tasks when connected to the DataReader.

Web Services have some interesting changes as well. First, there are changes to the way you access Web Services. The Add Web Reference dialog now lets you select Web Services from your local system, a Universal Description, Discovery, and Integration (UDDI) directory, or UDDI servers on your network. The Add Web Reference dialog makes working with Web Services simpler and improves not only the time required to access them, but reduces the tasks required to accomplish this.

Windows Server 2003 includes a new UDDI service, which provides a repository for storing information about Web Services. This will make it easy to hook up applications to Web Services that are referenced with the UDDI service, so your developers can easily find and consume Web Services on your network.

The Add Web Reference dialog has also changed, as shown in Figure 3. This is helpful because not only can you connect to the Web Service, but you can also change the name of the folder that contains the Web reference before it is configured.

Figure 3 Add Web Reference
Figure 3 Add Web Reference

One the client side, I found a new Windows Forms component that will also speed up the development process for Windows-based applications. The FolderBrowserDialog control provides a neat dialog from which you can easily select a folder. This allows you to drop this control on a form and wire it up with a few lines of code. The component also allows a user to create a new folder.

After you drop the component on a form, you can set several options and then display the dialog. The following code sets the starting (rootfolder) of the dialog to either the Desktop folder or MyComputer folder:

   If rdoDesktop.Checked Then
   FolderBrowserDialog1.RootFolder = _
       Environment.SpecialFolder.Desktop
   Else
   FolderBrowserDialog1.RootFolder = _
       Environment.SpecialFolder.MyComputer
   End If
The next few lines of code either turn the dialog's New Folder button on or off:
If rdoNewTrue.Checked Then
   FolderBrowserDialog1.ShowNewFolderButton = True
Else
   FolderBrowserDialog1.ShowNewFolderButton = False
End If
You can also control the description of the dialog by setting the Description property:
FolderBrowserDialog1.Description = "Select folder for files"
To display the dialog, you call the ShowDialog method. This method returns a dialog result that you can check to make sure the user selected a file:
If FolderBrowserDialog1.ShowDialog() = DialogResult.OK Then
Finally, you can retrieve the path selected by the user:
   StatusBar1.Text = FolderBrowserDialog1.SelectedPath
End If

The Environment.SpecialFolder enumeration returns an integer that represents a special folder on your system. You can use it to set the start folder for browsing, as shown earlier. This enumeration is also handy to use any time you need to allow the user to select one of these folders.

Also noteworthy is the new Enterprise Instrumentation Framework (EIF), which provides a framework to instrument your application. The vast majority of applications do not have good instrumentation because it's not part of the specification, and developers do not always take the time to implement it. The EIF makes this much easier to do, so hopefully more developers will instrument their applications.

To top off what I think are the cool features of the .NET Framework 1.1 release, you now get Oracle data providers. You can use these data providers right out of the box (or download) to connect to and work with an Oracle database without having to download, install, and configure them separately.

Now you've seen just a very few of the reasons to upgrade to Visual Studio .NET 2003. Since Visual Studio was first introduced, each new version has improved upon the previous one—that's why you upgrade. But it gets better: Visual Studio .NET 2003 and the .NET Framework 1.1 can be installed on the same system that continues to run Visual Studio .NET 2002. You don't have to throw out Visual Studio .NET 2002 just because you want to start using Visual Studio .NET 2003 for some projects.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencerworks for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


Security Briefs: Exploring S4U Kerberos Extensions in Windows Server 2003

Security Briefs
Exploring S4U Kerberos Extensions in Windows Server 2003
Keith Brown

Contents
Discovering Authorization Information
S4U2Self
The Problem of Delegation
Constrained Delegation or S4U2Proxy
Protocol Transition
Controls and Limitations
Conclusion

Building Web sites that provide services external to the corporate firewall is tricky. Usually it's not desirable to grant corporate domain accounts to external clients, and from a purely practical standpoint Kerberos does not work well over the Internet due to the typical configuration of client-side firewalls. This means that the site has to provide some form of authentication and authorization that is different from the methods built into Windows®.

To help deal with this and the other challenges I'll describe, Microsoft has implemented two extensions to Kerberos designed for use on servers. These extensions are collectively called Service-for-User (S4U). The goal is to allow server developers to program against the built-in security model that Windows provides, making use of the group-based authorization framework that administrators are already familiar with.

This column assumes a basic familiarity with Kerberos. For an online primer, see "Exploring Kerberos, the Protocol for Distributed Security in Windows 2000" in the August 1999 issue of Microsoft Systems Journal. For a more detailed explanation, see the book Programming Windows Security (Addison-Wesley, 2000). For the spec, see RFC 1510 at the IETF site (http://www.ietf.org/rfc/rfc1510.txt).

Discovering Authorization Information

The first extension I'll introduce helps a server discover the groups that a domain user belongs to. The authorization framework in Windows has become so complicated that it's virtually impossible for a server developer to manually discover the groups for a user. Global groups come from the user's home domain and may be nested. Universal groups can come from all over the user's home forest, are stored in the global catalog, and may be nested as well. Domain local groups are stored in the server's domain and may be nested. Local groups are stored on the server machine and are the easiest of the bunch to deal with. The SIDHistory feature in Windows 2000 means the resulting group expansion may need to be done multiple times for some users. Another feature called domain quarantine might mean removing some groups. Simply put, trying to manually figure out the group membership of a domain user is a developer's worst nightmare, and should be avoided.

The best way to get an authoritative list of groups for a user is to establish a logon for that user and look at the resulting token. During Kerberos authentication, the domain controllers perform all the heavy lifting. The only problem is, to perform Kerberos authentication on a client's behalf, the server needs to have the client's primary credentials, which basically means having access to the client's password or the client's ticket-granting ticket (TGT) and the corresponding session key. If the server knew the client's password, here's how simple it would be to get a token for a client named Alice, in the domain sales:

HANDLE hToken;
if (LogonUser("alice", "sales", alicesPassword, LOGON32_LOGON_NETWORK,
              0, &hToken)) {
  // hToken holds a token with all of Alice's groups
}
The obvious problem here is that the server must know the client's password, which is a severe breach of trust.

S4U2Self

The S4U solution in this case is for the server to go through the motions of Kerberos authentication and obtain a logon for the client, but without providing the client's credentials. Thus, you're not really authenticating the client in this case, only making the rounds to collect the group security identifiers (SIDs) for the client. To allow this to occur, Windows Server 2003 domain controllers accept a new type of Kerberos request, where the service requests a ticket from the client to itself, presenting its own credentials instead of the client's. This extension is called Service-for-User-to-Self (S4U2Self).

If the client and the service are in separate domains, this requires a bidirectional trust path between them because the service, acting on the client's behalf, must request tickets from the client's domain.

While the wire-level details are all rather complicated, the service developer need only call one function to start the ball rolling: LsaLogonUser. In spirit, this is similar to calling LogonUser as I have shown earlier, but without needing to provide the client's password. The result is a token that the service can use with functions like AccessCheck and CheckTokenMembership, as well as the new AuthZ family of authorization functions. This allows the service to perform access checks against security descriptors for objects that it manages.

To protect the client, LsaLogonUser normally returns a token with a special restriction. The token will have an impersonation level of Identify, which means that the service will not be able to open kernel objects while impersonating the client using this token. However, for services that are part of the trusted computing base (TCB)—for example, a service running as SYSTEM—LsaLogonUser will return a token with an impersonation level of Impersonate, allowing access to local kernel objects using the client's identity. This prevents an untrusted service from using an S4U2Self ticket to elevate its own local privileges.

Developers who want to use this feature need only call LsaLogonUser. However, LsaLogonUser is a difficult function to use. It expects 14 arguments, some of which are pointers to variable-length structures or handles to things that need opening first. Version 1.1 of the Microsoft® .NET Framework (in beta as of this writing) includes a new constructor on the WindowsIdentity class that tremendously simplifies this:

public WindowsIdentity(string userPrincipalName);

Figure 1 shows some sample C++ code that calls LsaLogonUser directly, requesting an S4U2Self logon.

Figure 1 Calling LsaLogonUser (Close Figure 1)
#define UNICODE
#define _UNICODE
#include <windows.h>
#include <ntsecapi.h>
#include <stdio.h>

#pragma comment(lib, "secur32.lib")

// some simple error handling functions
void _err(const wchar_t* fcn, DWORD err = GetLastError()) {
    wchar_t msg[256];
    if (!FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, 0, err,
        0, msg, sizeof msg / sizeof *msg, 0)) {
        wsprintf(msg, L"hr = 0x%08X", err);
    }
    wprintf(L"%s failed: %s\n", fcn, msg);
    exit(1);
}

void _nterr(const wchar_t* fcn, NTSTATUS s) {
    _err(fcn, LsaNtStatusToWinError(s));
}

void _checknterr(const wchar_t* fcn, NTSTATUS s) {
    if (s) {
        _nterr(fcn, s);
    }
    else {
        wprintf(L"%s succeeded.\n", fcn);
    }
}

// helper for initializing Kernel type ANSI strings
// from string literals
void _si(LSA_STRING* a, const char* b) {
    a->Length = (USHORT)(strlen(b) * sizeof(*b));
    a->MaximumLength = (USHORT)(a->Length + sizeof(*b));
    a->Buffer = const_cast<char*>(b);
}

HANDLE _s4uLogon(const wchar_t* pszUserPrincipalName) {
    // connect to the Local Security Authority
    // this will allow us to get a token via S4U2Self
    // (note we aren't in the TCB so we won't be able
    //  to use the resulting token to open Kernel objects)
    HANDLE hlsa;
    _checknterr(L"LsaConnectUntrusted",
        LsaConnectUntrusted(&hlsa));

    // look up the Kerb authentication provider's index
    LSA_STRING pkgName;
    _si(&pkgName, MICROSOFT_KERBEROS_NAME_A);
    ULONG authnPkg;
    _checknterr(L"LsaLookupAuthenticationPackage",
        LsaLookupAuthenticationPackage(hlsa,
            &pkgName, &authnPkg));

    const DWORD cchUPN = lstrlen(pszUserPrincipalName);
    const DWORD cbUPN = cchUPN * sizeof(wchar_t);

    // KERB_S4U_LOGON must be passed as a single
    // contiguous buffer that includes all strings,
    // otherwise LsaLogonUser will complain
    const DWORD cbLogon = sizeof(KERB_S4U_LOGON) + cbUPN;
    KERB_S4U_LOGON* s4uLogon =
        (KERB_S4U_LOGON*)calloc(cbLogon, 1);
    s4uLogon->MessageType = KerbS4ULogon;
    s4uLogon->ClientUpn.Buffer =
        (wchar_t*)((char*)s4uLogon + sizeof *s4uLogon);
    CopyMemory(s4uLogon->ClientUpn.Buffer,
        pszUserPrincipalName, cbUPN);
    s4uLogon->ClientUpn.Length        = (USHORT)cbUPN;
    s4uLogon->ClientUpn.MaximumLength = (USHORT)cbUPN;

    // this information is copied into the resulting token
    // note that SourceName is an 8 character ASCII buffer
    TOKEN_SOURCE tokenSource;
    AllocateLocallyUniqueId(&tokenSource.SourceIdentifier);
    strcpy(tokenSource.SourceName, "test");
    LSA_STRING originName;
    _si(&originName, "MSDN S4U Logon Sample");

    // finally, the call to LsaLogonUser,
    // asking for a Network style logon
    // using the S4U2Self Kerb extension
    void* profile = 0;
    DWORD cbProfile = 0;
    LUID logonId;
    HANDLE htok;
    QUOTA_LIMITS quotaLimits;
    NTSTATUS subStatus;
    _checknterr(L"LsaLogonUser",
        LsaLogonUser(hlsa,
            &originName,
            Network,
            authnPkg,
            s4uLogon, cbLogon,
            0,
            &tokenSource,
            &profile, &cbProfile,
            &logonId,
            &htok,
            &quotaLimits,
            &subStatus));

    // clean up
    free(s4uLogon);
    LsaFreeReturnBuffer(profile);
    LsaClose(hlsa);

    return htok;
}

void main() {
    HANDLE htok = _s4uLogon(L"alice@esec.com");
}

The Problem of Delegation

Delegation is another problem that has plagued developers. Even when a client is able to use Kerberos to authenticate with a service, the client's security context given to the service does not normally contain the client's network credentials. Imagine what could happen without this protection: a service, which may or may not be trusted in its own domain, could use a client's credentials from another domain to access resources it would normally not be able to access on its own. No client would ever want to use such a service, unless that service was highly trusted.

Windows 2000 introduced a feature called unconstrained delegation that allowed clients to delegate credentials to certain services that were designated as "trusted for delegation." In this case, unconstrained means that the service could use these delegated credentials anywhere on the network, all day long. Very few organizations actually used this feature, however, for two reasons. First, compromise of a service trusted for delegation means compromise of all the delegated credentials it holds, making it a virtual honeypot for attackers. Second, the local administrator of such a service must be highly trusted, lest he elevate his own privileges by misusing client credentials on the network. It was clear that this feature would be used only in a handful of rare situations.

Constrained Delegation or S4U2Proxy

The Active Directory® implementation in Windows Server 2003 has matured significantly since Windows 2000, and it supports a more reasonable delegation model known as constrained delegation. The idea is that a service may be allowed to delegate client credentials, but domain controllers will be much more selective about issuing such credentials. Thus, when service A requests Kerberos tickets for service B using delegated client credentials, the domain will consult a list to see if service A is allowed to delegate credentials to service B. In other words, the network administrator can control the scope of delegation.

Figure 2 Constrained Delegation
Figure 2 Constrained Delegation

An example of this would be a COM+ service hosting business logic. The service might need to access a remote database using the client's credentials. The network administrator can grant limited delegation privileges to the COM+ service, allowing it to delegate only to the database service. Figure 2 shows how this can be configured with the Active Directory Users and Computers tool. In this case, the COM+ service runs under the built-in Network Service account on a machine called AppServer, and the remote database is on a machine called DataServer. Contrast this to Windows 2000, where the only choice for delegation was a checkbox that turned unconstrained delegation on or off, as shown in Figure 3.

Figure 3 Unconstrained Delegation
Figure 3 Unconstrained Delegation

Kerberos, as defined by RFC 1510, does provide a constrained delegation feature known as a proxy ticket, but the client must request these tickets on the service's behalf. The Microsoft extension to Kerberos, known as S4U2Proxy (Service-for-User-to-Proxy), allows the service to obtain tickets on the client's behalf based on configuration settings in Active Directory. With S4U2Proxy, a service can obtain tickets to some other service (the proxy) for a user. This means the client must trust the directory (and its administrators) because the client no longer has control over whether her credentials may be delegated.

Technically, the biggest difference between normal Kerberos proxy delegation and S4U2Proxy is that in standard Kerberos the client's TGT is required to obtain proxy tickets. In S4U2Proxy, since the service is requesting the tickets on the client's behalf and the service does not have the client's TGT, the service submits a service ticket instead, which will be either the ticket the client presented to the service during normal Kerberos authentication or a ticket obtained on the client's behalf via S4U2Self. In the latter case, the client may not even have been authenticated using Kerberos, which may seem a bit odd. This particular case is known as protocol transition.

Protocol Transition

A common problem with externally facing services in Windows 2000 is client authentication and authorization. Since Kerberos authentication isn't usually an option over the Internet, these services must often use other authentication techniques such as Secure Sockets Layer (SSL) with client certificates, hand-rolled forms-based solutions, Microsoft Passport, or other proprietary authentication protocols such as SecurID from RSA Security. Once the service determines the client's identity, the challenge is to convert this knowledge into real Windows credentials that can be used to access local resources and remote services on the client's behalf.

Once again, there are two solutions to this problem in Windows 2000: either the service knows the password for an account representing the client and uses this password to obtain a Kerberos logon, or the service must scrape through Active Directory manually to figure out the group memberships for the user. Because the service couldn't use Kerberos to authenticate its client, it is effectively penalized by not being able to easily make authorization decisions based on settings in Active Directory.

Protocol Transition uses both of the S4U Kerberos extensions I've described to make it possible for a service in this position to transition from a proprietary authentication protocol to the Microsoft Kerberos implementation on the back end. Figure 4 shows what protocol transition looks like.

Figure 4 Protocol Transitioning
Figure 4 Protocol Transitioning

Developers can use this technique by authenticating the client using whatever technique makes sense, then calling LsaLogonUser to obtain a token for the user via the S4U2Self extension that I discussed earlier. The developer writes code to impersonate the resulting token and makes authenticated requests to any of the back-end services to which it is allowed to delegate. The back-end services will think the client is accessing them—not the front-end service—and will receive a security context for the client, along with all groups and privileges that the client would have if she had authenticated directly with the back-end service. Thus the front-end server acts as a trusted point of authentication protocol transition. The network administrator controls which back-end services trust the front-end service via the constrained delegation settings in Active Directory.

Note that the network administrator must explicitly allow the server to use protocol transition by granting constrained delegation using "any authentication protocol" on the front end.

Controls and Limitations

Impersonation and delegation of a client's credentials is a sensitive matter, and several controls are maintained in Active Directory to prevent abuse of these extensions. A service that wants to use S4U2Self to obtain a token for a client must be granted permission to enumerate the groups for that client. The network administrator controls this permission via the access control list on the Active Directory user object attribute known as Token-Groups-Global-And-Universal, which isn't accessible via the Active Directory Users and Computers snap-in, but can be accessed interactively using ADSIEdit or programmatically via ADSI.

When a service requests an S4U2Self ticket for a client, if protocol transition has been enabled for that service in the directory, the resulting ticket will have the forwardable flag set. This controls whether the service can use this ticket to further obtain S4U2Proxy tickets to delegate the client's credentials to other services. If the client's account has been marked "Sensitive and cannot be delegated," the forwardable flag will not be set, and delegation will not be allowed. Even if the forwardable flag is set, S4U2Proxy tickets will only be issued for services that the requesting service is allowed to delegate to (refer back to Figure 2 to take a look at the list in Active Directory).

S4U2Proxy tickets will not work if the path from the client's domain to the target resource domain spans more than two forests. The reason for this has to do with the way cross-forest security identifier filtering works.

As I mentioned earlier, LsaLogonUser behaves differently depending on whether the caller has the TCB privilege. If the caller enables the TCB privilege before calling LsaLogonUser, it will return a token that can be used to access local resources on the machine; otherwise, this will be disallowed via the impersonation level on the token. This is orthogonal to whether the forwardable flag is set, so technically you can end up with a token that you can impersonate and use for delegation to remote services, but while impersonating you will be denied access to all local kernel objects. An example of this would be a service running as NETWORK SERVICE (as opposed to SYSTEM) that has been granted the right to use constrained delegation.

Note that for brevity I've mentioned permissions that a service must be granted. What I mean by this is that the account the service is running under must be granted the permissions. For example, if the service is running as Bob, Bob must be granted the permissions. More subtly, if the service is running as NETWORK SERVICE or SYSTEM, it is running on behalf of the machine itself, so the machine account must be granted the permissions.

Conclusion

Tightly coupling authorization with authentication has been both a blessing and a curse for Microsoft. The benefits are clear. As the client obtains Kerberos tickets, domain controllers inject group SIDs into the tickets, which eliminates the network round-trips that would be needed to get to a separate authorization service. The administration model is also simplified. The downside is that the only way to get authoritative authorization information is via Kerberos authentication, and extensions like these don't make it any easier to interoperate with other implementations of Kerberos. In addition, the bidirectional trust that must be established between forests for these features to work across forest boundaries is another potential stumbling block.

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brownis a member of the technical staff at DevelopMentor, where he spends his time researching, writing, teaching, and spreading the word about security. He is the author of Programming Windows Security (Addison Wesley, 2000). Reach Keith at http://www.develop.com/kbrown.


Wicked Code: Supporting Database Cache Dependencies in ASP.NET

Wicked Code
Supporting Database Cache Dependencies in ASP.NET
Jeff Prosise

Code download available at: WickedCode0304.exe (135KB)
Contents
Database Dependencies in Action
How Database Dependencies Work
Server Farms

Developers love the ASP.NET application cache. One reason they love it is that ASP.NET lets them create dependencies between items placed in the cache and files in the file system. If a file targeted by a dependency changes, ASP.NET automatically removes dependent items from the cache. Combined with cache removal callbacks—notifications broadcast to interested parties when cached items are removed—cache dependencies are a boon to developers seeking to maximize performance by minimizing time-consuming file accesses because they permit file data to be cached without fear of it becoming stale.

As awesome as cache dependencies are, in ASP.NET version 1.0 they lack one critical feature that, if present, would qualify them as a developer's dream come true: support for database entities. In real life, most Web apps fetch data from databases, not files. But while ASP.NET is perfectly willing to link cached items to files, it is incapable of linking cached items to database entities. In other words, you can read the contents of a file into a DataSet, cache the DataSet, and have the DataSet automatically removed from the cache if the file it was initialized from changes. But you can't initialize a DataSet with a database query, cache the DataSet, and have the DataSet automatically discarded if the database changes. That's too bad because too many database accesses, like too much file I/O, is a performance killer.

The fact that ASP.NET doesn't support database dependencies doesn't mean database dependencies are impossible to achieve. This installment of Wicked Code presents a technique for extending the ASP.NET application cache to support database dependencies. It involves database triggers and extended stored procedures. Though the implementation presented here works exclusively with Microsoft® SQL Server™, the general technique is applicable to any database that supports triggers and user-defined procedures that interact with the file system.

Database Dependencies in Action

Let's begin with a demonstration. Figure 1 contains the source code for an ASP.NET page that displays randomly selected quotations from a SQL Server database named Quotes. To create the database, run the installation script shown in abbreviated form in Figure 2. The complete script is included in the downloadable zip file that accompanies this column. You can execute the script inside the SQL Server Query Analyzer or from the command line with an OSQL command.) Each time the page is fetched, Page_Load initializes a DataSet with all the records in the database's Quotations table, randomly selects a record from DataSet, and writes it to the page. Press F5 a few times and you'll see a random variety of quotations from some famous (and not-so-famous) people, as shown in Figure 3.

Figure 2 Quotes-1.sql (Abbreviated) (Close Figure 2)
CREATE DATABASE Quotes
GO

USE Quotes
GO

CREATE TABLE Quotations (
  Quotation varchar(256) NOT NULL,
  Author    varchar(64)  NOT NULL
)
GO

INSERT INTO Quotations (Quotation, Author) VALUES (
  'Give me chastity and continence, but not yet.',
  'Saint Augustine'
)
  •••
INSERT INTO Quotations (Quotation, Author) VALUES (
  '640K ought to be enough for anybody.',
  'William Gates III'
)
GO
Figure 1 DumbDBQuotes.aspx (Close Figure 1)
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>

<html>
  <body>
    <h3><asp:Label ID="Quotation" RunAt="server" /></h3>
    <i><asp:Label ID="Author" RunAt="server" /></i>
  </body>
</html>

<script language="C#" runat="server">
void Page_Load (Object sender, EventArgs e)
{
    SqlDataAdapter adapter = new SqlDataAdapter (
        "SELECT * FROM Quotations",
        "server=localhost;database=quotes;uid=sa;pwd="
    );

    DataSet ds = new DataSet ();
    adapter.Fill (ds, "Quotations");
    DataTable table = ds.Tables["Quotations"];

    Random rand = new Random ();
    int index = rand.Next (0, table.Rows.Count);
    DataRow row = table.Rows[index];

    Quotation.Text = (string) row["Quotation"];
    Author.Text = (string) row["Author"];
}
</script>
Figure 3 Random Quote
Figure 3 Random Quote

The page is called DumbDBQuotes.aspx for a reason. It's not very smart considering it queries the database each time it's requested. Accessing a database on every page access—especially a database hosted on a remote server—is a guaranteed way to build an application that won't scale.

The ASP.NET application cache is the solution to the problem of too many database accesses. If the DataSet were cached, it could be fetched directly from memory—that is, from the cache—thereby eliminating the redundant database accesses. It's easy enough to cache a DataSet; the application cache accepts instances of any type that derives from System.Object. In the Microsoft .NET Framework, that means instances of any managed type, including DataSet. The problem is that if you cache a DataSet and the database changes underneath it, you serve stale data to your users. You could implement a solution that requeries the database periodically, but the ideal solution is the one that requires no polling and that delivers fresh data from the data source the moment that data becomes available.

Take a look at Figure 4 and Figure 5, which contain the source code for a smarter quotes application. SmartDBQuotes.aspx doesn't retrieve quotations from the database; it gets them from the application cache. Global.asax primes the cache and refreshes it if the database changes. Here are directions for taking them for a test drive:

  1. Create a subdirectory named AspNetSql in the root directory of your Web server's C: drive. Inside AspNetSql, create a zero-byte file named Quotes.Quotations. Make sure Everyone, or at least SYSTEM and ASPNET (a special account created when ASP.NET is installed), has access to Quotes.Quotations.
  2. Copy XSP.dll, which is included in this column's downloadable code sample, into the SQL Server binn directory (for example, C:\Program Files\Microsoft SQL Server\MSSQL\Binn) or to any location on your Web server that Windows will automatically search for DLLs (for example, C:\Windows\System32).
  3. Rebuild the database using the modified script in Figure 6.
  4. Deploy Global.asax and SmartDBQuotes.aspx to a virtual directory on your Web server (for example, wwwroot).
  5. Request SmartDBQuotes.aspx in your browser. Refresh the page a few times until the quote "The use of COBOL cripples the mind; its teaching should therefore be regarded as a criminal offense" appears.
  6. Use SQL Server Enterprise Manager or the tool of your choice to modify the quotation in the Quotes database's Quotations table. Change it to read "The use of Visual Basic® enriches the mind; its teaching should therefore not be regarded as a criminal offense." Then refresh the page until the modified quotation appears. Observe that the new quotation appears, not the old one, even though the query results are now being stored in the application cache.

Figure 6 Quotes-2.sql (Abbreviated) (Close Figure 6)
USE master
EXEC sp_addextendedproc 'xsp_UpdateSignalFile', 'XSP.dll'
GRANT EXECUTE ON xsp_UpdateSignalFile TO PUBLIC
GO

CREATE DATABASE Quotes
GO

USE Quotes
GO

CREATE TABLE Quotations (
  Quotation varchar(256) NOT NULL,
  Author    varchar(64)  NOT NULL
)
GO

INSERT INTO Quotations (Quotation, Author) VALUES (
  'Give me chastity and continence, but not yet.',
  'Saint Augustine'
)
  •••
INSERT INTO Quotations (Quotation, Author) VALUES (
  '640K ought to be enough for anybody.',
  'William Gates III'
)
GO

CREATE TRIGGER DataChanged ON Quotations FOR INSERT, UPDATE, DELETE
AS EXEC master..xsp_UpdateSignalFile 'Quotes.Quotations'
GO
Figure 5 SmartDBQuotes.aspx (Close Figure 5)
<%@ Import Namespace="System.Data" %>

<html>
  <body>
    <h3><asp:Label ID="Quotation" RunAt="server" /></h3>
    <i><asp:Label ID="Author" RunAt="server" /></i>
  </body>
</html>

<script language="C#" runat="server">
void Page_Load (Object sender, EventArgs e)
{
    DataSet ds = (DataSet) Cache["Quotes"];

    if (ds != null) {
        // Display a randomly selected quotation
        DataTable table = ds.Tables["Quotations"];

        Random rand = new Random ();
        int index = rand.Next (0, table.Rows.Count);
        DataRow row = table.Rows[index];

        Quotation.Text = (string) row["Quotation"];
        Author.Text = (string) row["Author"];
    }
    else {
        // If quotes is null, this request arrived after the
        // DataSet was removed from the cache and before a new
        // DataSet was inserted. Tell the user the server is
        // busy; a page refresh should solve the problem.
        Quotation.Text = "Server busy";
    }
}
</script>
Figure 4 Global.asax (Close Figure 4)
<%@ Import NameSpace="System.Data" %>
<%@ Import NameSpace="System.Data.SqlClient" %>

<script language="C#" runat="server">
static Cache _cache = null;

void Application_Start ()
{
    _cache = Context.Cache; // Save reference for later

    //
    // Query the database and cache the resulting DataSet.
    //
    RefreshCache (null, null, 0);
}

static void RefreshCache (string key, object item,
    CacheItemRemovedReason reason)
{
    //
    // Query the database.
    //
    SqlDataAdapter adapter = new SqlDataAdapter (
        "SELECT * FROM Quotations",
        "server=localhost;database=quotes;uid=sa;pwd="
    );

    DataSet ds = new DataSet ();
    adapter.Fill (ds, "Quotations");

    //
    // Add the DataSet to the application cache.
    //
    _cache.Insert (
        "Quotes",
        ds,
        new CacheDependency ("C:\\AspNetSql\\Quotes.Quotations"),
        Cache.NoAbsoluteExpiration,
        Cache.NoSlidingExpiration,
        CacheItemPriority.Default,
        new CacheItemRemovedCallback (RefreshCache)
    );
}
</script>

You just demonstrated that the ASP.NET application cache can be combined with database dependencies to produce high-volume, data-driven applications. The question is how? How was a link formed between the cached DataSet and the database, and how scalable is the solution?

How Database Dependencies Work

On the outside, both DumbDBQuotes.aspx and SmartDBQuotes.aspx look the same, producing identical output. On the inside, they could hardly be more different. The former performs a database access every time it's requested; the latter fetches data from the application cache. Moreover, SmartDBQuotes.aspx uses a database dependency to ensure that if the database changes, the cached data changes, too. If the database doesn't change, the database is queried just once during the lifetime of the application. If the database changes, one more query updates the cache.

Figure 7 Database Dependencies
Figure 7 Database Dependencies

Figure 7 illustrates how the database dependency works. When it places the DataSet in the cache, Global.asax creates a file-system dependency between the DataSet and a file named Quotes.Quotations in the C:\AspNetSql directory. Quotes.Quotations is a zero-byte signal file—a file whose only purpose is to trigger the ASP.NET application cache-removal logic. Here's the statement in Global.asax that creates the CacheDependency object linking the DataSet to Quotes.Quotations:

new CacheDependency ("C:\\AspNetSql\\Quotes.Quotations")
Global.asax also registers its own RefreshCache method to be called when the DataSet is removed from the cache—that is, when the signal file changes:
new CacheItemRemovedCallback (RefreshCache)

RefreshCache's job is to query the database and place the resulting DataSet in the application cache. It's called once when the application starts up and again when—or if—the DataSet is removed from the cache.

That's half of the equation. The other half involves the database. The revised database installation script in Figure 6 adds an insert/update/delete trigger to the database's Quotations table:

CREATE TRIGGER DataChanged ON Quotations
FOR INSERT, UPDATE, DELETE
AS EXEC master..xsp_UpdateSignalFile 'Quotes.Quotations'
GO
The trigger fires when records are added to or deleted from the table and when records change. What does the trigger do? It calls an extended stored procedure—the SQL Server euphemism for code in a Win32® DLL—named xsp_UpdateSignalFile. The extended stored procedure, in turn, uses the Win32 CreateFile function to update Quotes.Quotations' time stamp.

The cached DataSet's lifetime is tied to Quotes.Quotations using an ordinary file-system cache dependency; updating the Quotations table causes a database trigger to fire; and the trigger calls an extended stored procedure that "updates" Quotes.Quotations, prompting ASP.NET to remove the DataSet from the application cache and call Global.asax's RefreshCache method, which then performs a brand new database query and starts the whole process all over again.

The final piece of the puzzle is the extended stored procedure. It's housed in XSP.dll, the DLL that you installed earlier. I wrote XSP.dll in unmanaged C++ using Visual C++® 6.0. Its source code appears in Figure 8. The path to the signal file—C:\AspNetSql—is hardcoded into the DLL, but you can change that if you'd like and make it an input parameter just like the file name.

Figure 8 XSP.cpp (Close Figure 8)
#include <windows.h>
#include <srv.h>

///////////////////////////////////////////////////////////////////////
// Entry point

extern "C" BOOL WINAPI DllMain (HINSTANCE hInstance, DWORD dwReason,
    LPVOID lpReserved)
{
    return TRUE;
}

///////////////////////////////////////////////////////////////////////
// Exported functions

extern "C" __declspec (dllexport)
ULONG __GetXpVersion ()
{
    return ODS_VERSION;
}

extern "C" __declspec (dllexport)
SRVRETCODE xsp_UpdateSignalFile (SRV_PROC *srvproc)
{
    //
    // Make sure an input parameter is present.
    //
    if (srv_rpcparams (srvproc) == 0)
        return -1;

    //
    // Extract the file name from the input parameter.
    //
    BYTE bType;
    char file[256];
    ULONG ulMaxLen = sizeof (file);
    ULONG ulActualLen;
    BOOL fNull;

    if (srv_paraminfo (srvproc, 1, &bType, &ulMaxLen, &ulActualLen,
        (BYTE*) file, &fNull) == FAIL)
        return -1;

    if (bType != SRVBIGCHAR && bType != SRVBIGVARCHAR)
        return -1;

    file[ulActualLen] = 0;

    //
    // Update the file's time stamp.
    //
    char path[288] = "C:\\AspNetSql\\";
    lstrcat (path, file);

    HANDLE hFile = CreateFile (path, GENERIC_WRITE, 0, NULL,
        CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

    if (hFile != INVALID_HANDLE_VALUE)
        CloseHandle (hFile);

    return 0;
}

Extended stored procedures must be installed before they're used. The following statements in the SQL installation script that you executed install xsp_UpdateSignalFile in the master database and grant execute permission to all comers:

USE master
EXEC sp_addextendedproc 'xsp_UpdateSignalFile', 'XSP.dll'
GRANT EXECUTE ON xsp_UpdateSignalFile TO PUBLIC
GO
Why write a custom extended stored procedure to update a file's time stamp when a built-in extended stored procedure such as xp_cmdshell could be used instead? The reason is security—xp_cmdshell can be used for all sorts of malicious purposes, while xsp_UpdateSignalFile cannot. Because xsp_UpdateSignalFile does little more than call the Windows CreateFile function, it is also more efficient than xp_cmdshell.

Server Farms

SmartDBQuotes.aspx and friends work great if the Web server and the database server live on the same machine, but what if the database is installed on a different machine? And what about Web farms? Would a change notification mechanism based on database triggers, extended stored procedures, and file-system dependencies be compatible with multiserver installations?

You bet. Under the hood, ASP.NET cache dependencies that are based on the file system rely on Win32 file change notifications. And Win32 file change notifications support Universal Naming Convention (UNC) path names. To take advantage of database cache dependencies on Web farms, let the signal file reside on the database server, as shown in Figure 9. Then pass CacheDependency's constructor a UNC path name specifying the signal file's network address:

new CacheDependency
(@"\\ServerName\AspNetSql\Quotes.Quotations"),

Figure 9 Signal File on Database Server
Figure 9 Signal File on Database Server

The greatest obstacle to creating dependencies that target remote files is security. By default, the ASP.NET worker process runs as ASPNET when paired with Microsoft Internet Information Services (IIS) 5.0 and when configured to run in compatibility mode under IIS 6.0. ASPNET is a local account that can't authenticate on remote machines. Without configuration changes, attempting to create a cache dependency with a UNC path name produces an access denied error—even if you give Everyone access to the remote share.

Several solutions exist. One is to configure ASP.NET to use a domain account that can be authenticated on the database server. That change is easy enough to accomplish: you simply specify the account's user name and password in the <processModel> section of each Web server's Machine.config. Many companies, however, have security policies that prevent passwords from being stored in plaintext configuration files. If that's true of your company, but you'd still like to run ASP.NET using a domain account, you can either upgrade to version 1.1 of the .NET Framework (which allows worker process credentials to be encrypted and stored securely in the registry) or download a hotfix for version 1.0 that does the same. You'll find information about the hotfix at Stronger Credentials for processModel, identity, and sessionState.

A variation on this technique involves setting up identical local accounts (using the same user name and password) on both machines and configuring ASP.NET to run as that identical local account on the Web server.

Another solution to the problem of authenticating on a back-end database server containing a signal file is to upgrade to Windows Server 2003. The latest addition to the Windows Server family comes with IIS 6.0, which allows the ASP.NET worker process to run using the identity of Network Service. Unlike ASPNET, Network Service can perform authentication on remote machines. Or you could pull an old trick out of the ASP playbook and access the database through a COM+ component running on the Web server and configure the component to run as a principal that has network credentials.

However you choose to make the remote signal file accessible to your ASP.NET app, the bottom line is that combining database cache dependencies with UNC path names delivers a scalable solution that works equally well for Web farms and single-server installations. That's good news for developers who are using ASP.NET to build high-volume, data-driven applications—and good news for users as well.

Send your questions and comments to Jeff at wicked@microsoft.com.

Jeff Prosiseis the author of several books, including Programming Microsoft .NET and Programming Windows with MFC, both from Microsoft Press. He is also a cofounder of Wintellect, a consulting and training firm that specializes in the Microsoft .NET Framework.


C++ Q&amp;A: Desktop Location, sscanf Equivalents in C#, and More

C++ Q&A
Desktop Location, sscanf Equivalents in C#, and More
Paul DiLascia

Code download available at: CQA0304.exe (154KB)

Q Can you tell me why the following C# form doesn't run where I want it to on my desktop?

public class MyForm : Form
{
  MyForm()
  {
    Text = "Why doesn't this form go where I want it to go?" ;
    DesktopLocation = new System.Drawing.Point (500, 500) ;
  }
  static void Main()
  {
    Run(new MyForm());
  }
}
The title bar text in the C# form is fine, but the DesktopLocation seems to be ignored.

Doug Harrington

A You can set DesktopLocation all you want, but the Microsoft® .NET Framework won't pay attention unless you also set StartPosition in the following way:

StartPosition = FormStartPosition.Manual;

As Mensa members might guess, Form.StartPosition tells the .NET Framework where to start your form. The possible values are CenterParent, CenterScreen, Manual, WindowsDefaultBounds, and WindowsDefaultLocation. Even non-Mensa folks can guess what these mean. By default, StartPosition is WindowsDefaultLocation, the CreateWindow equivalent of CW_USEDEFAULT.

Some of you might wonder, what's the difference between Location and DesktopLocation? Answer: Location is relative to the screen, whereas DesktopLocation is relative to the desktop work area—that is, the screen minus the task bar. That's something to consider if you suspect your user may be crazy enough to dock her task bar on the top of the screen (I hear this is more common in the Southern Hemisphere).

With a top-docked task bar, setting Form.Location = (0,0) results in a partially hidden form, whereas DesktopLocation = (0,0) makes everything first rate. DesktopLocation also works on multiple-monitor systems. (If you aren't programming on a 9 × 12 wall-sized array of plasma screens, you're nobody.) And in case you're wondering where to find DesktopLocation in the Properties window in Visual Studio®—it's not there. You can only set DesktopLocation from code. What a concept!

Q If I want to parse a hex string in C++, I can use sscanf like so:

char hex[] = "0xFA10";
int i;
sscanf( hex, "%x", &i );
But is there a way for me to do this in the .NET Framework? Is there an equivalent to sscanf?

Gary Miller

A There's no direct equivalent of sscanf in the .NET Framework; however, you can use Int32.Parse to parse a hex number like so:

Int32 i = Int32.Parse("0FA10", NumberStyles.HexNumber);

Note that in the .NET Framework, you must omit the leading "0x". HexNumber is actually a composite style, equivalent to the following flags: AllowLeadingWhite | AllowTrailingWhite | AllowHexSpecifier. In general, the Framework does parsing with—what else—a Parse method. Many classes in the Framework have one. There's Decimal.Parse and Double.Parse for decimals and doubles, IPAddress.Parse for IP addresses, and DateTime.Parse for dates. My favorite is Enum.Parse, which uses reflection to convert the text name of any Enum to its actual enum value. For example, the following code

Console.WriteLine("val={0:x}.",
   Enum.Parse(typeof(NumberStyles),
   "AllowHexSpecifier"));
yields the line:
val=00000200.

To see which classes have Parse functions, open your MSDN® disc to the Index, select ".NET Framework" as your docset filter, and then type "Parse".

Q I have a small application that I distribute over the Internet. I've seen programs that display a message when a new version is available and ask the user if he wants to upgrade. How can I implement a version checker like that for my own app?

Shika Kataransky

A I'll answer in just a moment, but first let me use my little soapbox to warn against one of my pet peeves: programs that display annoying messages that require action. In particular, I hate programs that ask whether I want to upgrade. I always answer no and check the "stop bothering me" checkbox (hopefully there is one). There's nothing wrong with informing users that a new version is out, but please do it in a way that doesn't require action—except, of course, to get the new version.

Now that I've scared you away, there are many ways to implement the Web-based version checking. The February 2003 issue of MSDN Magazine has an article by Jason Clark that describes a whole new protocol called BITS for just this purpose (see BITS: Write Auto-Updating Apps with .NET and the Background Intelligent Transfer Service API).

On the other hand, if all you really want is something that is quick and simple, you can store your current version number in a text file on your Web site and download it via FTP. Downloading is easy with the Windows® Internet API, also known as WinInet. Just open a connection, open an FTP session, open the file, and read. Here's the sequence in a nutshell:

HINTERNET h = InternetOpen(...);
HINTERNET hftp = InternetConnect(..,INTERNET_SERVICE_FTP,..);
HINTERNET hftpfile = FtpOpenFile(...);
InternetReadFile(...);

As with all things, beauty dwells in the details, so let's dive deeper. I wrote a class called CWebVersion (see Figure 1) that encapsulates all you need for a simple over-the-Web version checker. You can use it in the following manner:

if (CWebVersion::Online()) {
  CWebVersion ver("ftp.mysite.com");
  if (ver.ReadVersion("MyVersionNumber.txt")) {
    DWORD maj = ver.dwVersionMS;
    DWORD min = ver.dwVersionLS;
  }
}

Figure 1 WebVersion (Close Figure 1)

WebVersion.h
////////////////////////////////////////////////////////////////
// MSDN Magazine — April 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compile with Visual Studio .NET on Windows XP. Tab size=3.
//
#pragma once

//////////////////
// This class encapsulates over-the-Web version checking. It expects a
// text version file that contains four numbers separated by commas, the
// same format for FILEVERSION and PRODUCTVERSION in VS_VERSION_INFO.
// ReadVersion reads these numbers into dwVersionMS and dwVersionLS.
//
class CWebVersion {
protected:
   enum { BUFSIZE = 64 };
   LPCTSTR m_lpServer;                  // server name
   DWORD   m_dwError;                   // most recent error code
   TCHAR   m_errInfo[256];              // extended error info
   TCHAR   m_version[BUFSIZ];           // version number as text
   void    SaveErrorInfo();             // helper to save error info

public:
   DWORD dwVersionMS;      // version number: most-sig 32 bits
   DWORD dwVersionLS;      // version number: least-sig 32 bits

   CWebVersion(LPCTSTR server) : m_lpServer(server) { }
   ~CWebVersion() { }

   static  BOOL Online();
   BOOL    ReadVersion(LPCTSTR lpFileName);
   LPCTSTR GetVersionText()       { return m_version; }
   DWORD   GetError()             { return m_dwError; }
   LPCTSTR GetExtendedErrorInfo() { return m_errInfo; }
};
WebVersion.cpp
////////////////////////////////////////////////////////////////
// MSDN Magazine — April 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "stdafx.h"
#include "WebVersion.h"
#include "InetHandle.h"

//////////////////
// Check if connected to Internet.
//
BOOL CWebVersion::Online()
{
    DWORD dwState = 0;
    DWORD dwSize = sizeof(DWORD);
    return InternetQueryOption(NULL,
       INTERNET_OPTION_CONNECTED_STATE, &dwState, &dwSize)
       && (dwState & INTERNET_STATE_CONNECTED);
}

//////////////////
// Read version number as string into buffer
//
BOOL CWebVersion::ReadVersion(LPCTSTR lpFileName)
{
   CInternetHandle hInternet;
   CInternetHandle hFtpSession;
   CInternetHandle hFtpFile;

   m_version[0] = 0;
   m_dwError=0;                         // assume success
   m_errInfo[0]=0;                      // ..

   DWORD nRead=0;
   hInternet = InternetOpen(NULL, INTERNET_OPEN_TYPE_DIRECT, NULL,
                            NULL, 0);
   if (hInternet!=NULL) {
      hFtpSession = InternetConnect(hInternet, m_lpServer,
         INTERNET_DEFAULT_FTP_PORT, NULL, NULL, INTERNET_SERVICE_FTP,
         0, NULL);

      if (hFtpSession!=NULL) {
         hFtpFile = FtpOpenFile(hFtpSession, lpFileName,
            GENERIC_READ, FTP_TRANSFER_TYPE_ASCII, NULL);

         if (hFtpFile!=NULL) {
            InternetReadFile(hFtpFile, m_version, BUFSIZE, &nRead);
            if (nRead>0) {
               m_version[nRead] = 0;
               int Mhi,Mlo,mhi,mlo;
               _stscanf(m_version, "%x,%x,%x,%x", &Mhi, &Mlo, &mhi, &mlo);
               dwVersionMS = MAKELONG(Mlo,Mhi);
               dwVersionLS = MAKELONG(mlo,mhi);
               return TRUE;
            }
         }
      }
   }

   // Failed: save error code and extended error info if any.
   m_dwError = GetLastError();
   if (m_dwError==ERROR_INTERNET_EXTENDED_ERROR) {
      DWORD dwErr;
      DWORD len = sizeof(m_errInfo)/sizeof(m_errInfo[0]);
      InternetGetLastResponseInfo(&dwErr, m_errInfo, &len);
   }

   return FALSE;
}

CWebVersion::Online first checks to see if the user is online. I'll let you decide what to do when the user is offline—my recommendation is to do nothing, or display a "Check for latest version" button so users can explicitly initiate the connection. (The WinInet function for this is InternetAttemptConnect.) Whatever you do, don't connect without asking. Many people still use modems in order to surf the Web.

Assuming your application has access to a connection, all you have to do to read the version number is instantiate a CWebVersion object, passing the address of your FTP server, and call CWebVersion::ReadVersion with the name of your version file. ReadVersion expects a one-line text file that looks something like this:

4,3,0,0
That is, four numbers separated by commas, the same format as the FILEVERSION and PRODUCTVERSION fields in your VS_VERSION_INFO resource. ReadVersion reads the numbers into two 32-bit DWORDs, dwVersionMS and dwVersionLS. Windows uses 64-bit version numbers, so you can release 18,446,744,070,000,000,000 versions before cycling. That's 18 quintillion in the U.S., but only 18 trillion in Europe. It would take 8 million programmers 75,000 years to write that many versions, if they released a new one every second. You better hurry up and hone up your typing skills!

Figure 2 shows a test program called GetVersion that uses CWebVersion to read a version number over FTP; Figure 3 shows the program running.

Figure 2 GetVersion.cpp (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine — April 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "stdafx.h"
#include "WebVersion.h"

////////////////
// This program shows how to use CWebVersion, a class that reads a
// version number from your Web site using FTP. You can use it to
// test-read your own version number before adding CWebVersion to your
// app. To use it, open a command window and type:
//
// GetVersion <version-filename> <ftp.my.server.name>
//

LPCTSTR GetInetErrorName(DWORD err);

int _tmain(int argc, _TCHAR* argv[])
{
   if (argc != 3) {
      printf("\nGetVersion 1-1-2003 Paul DiLascia\n");
      printf("Illustrates how to retrieve a program version
             number from the web.\n");
      printf("usage: GetVersion filename server\n");
      printf(" filename = name of ASCII file containing version
             number\n");
      printf(" server   = name of FTP server, eg, ftp.mysite.com\n");
      return -1;
   }

   if (CWebVersion::Online()) {
      CWebVersion webver(argv[2]);
      if (webver.ReadVersion(argv[1])) {
         printf("Major version = %08x\n", webver.dwVersionMS);
         printf("Minor version = %08x\n", webver.dwVersionLS);
         printf("Text = %s\n", webver.GetVersionText());

      } else {
         DWORD err = webver.GetError();
         printf("%s = %d\n", GetInetErrorName(err), err);
         if (err==ERROR_INTERNET_EXTENDED_ERROR) {
            printf("%s\n", webver.GetExtendedErrorInfo());
         }
      }
   } else {
      printf("Can't get version: not online\n");
   }
   return 0;
}

//////////////////
// Handy helper function to get name of ERROR_INTERNET_XXX code.
// (Debug only)
//
LPCTSTR GetInetErrorName(DWORD err)
{
   static struct {
      DWORD err;
      LPCTSTR name;
   } errors[] = {
      { ERROR_INTERNET_OUT_OF_HANDLES,_T("ERROR_INTERNET_OUT_OF_HANDLES") },
      { ERROR_INTERNET_TIMEOUT,_T("ERROR_INTERNET_TIMEOUT") },
         •••
         // more
      { 0, NULL }
   };

   for (int i=0; errors[i].name; i++) {
      if (err==errors[i].err) {
         return errors[i].name;
      }
   }
   return _T("Unknown error");
}
Figure 3 GetVersion in Action
Figure 3 GetVersion in Action

How does CWebVersion work? Most of the functions are self-explanatory; however, a few details merit explanation. To see if the system is connected, CWebVersion::Online calls a WinInet function InternetQueryOption to get the INTERNET_OPTION_CONNECTED_STATE. If the state returned has the INTERNET_STATE_CONNECTED flag, you're live. To access the Internet and read the version file, CWebVersion uses another home-grown class, CInternetHandle, which encapsulates HINTERNET (see Figure 4). CInternetHandle's raison d'être is to make error-checking easier. If a WinInet function fails, GetLastError tells why. But CloseInternetHandle clears the error code, so you have to call GetLastError before CloseInternetHandle. You end up with code that looks something like Figure 5. With CInternetHandle, the code is much cleaner: ReadVersion only has to check the error at the end. C++ automatically closes all the CInternetHandles as flow control leaves your function.

Figure 5 yuck.cpp (Close Figure 5)
////////////////////////////////////////////////////////////////
// This is what your code would look like without CInternetHandle.
// You have to call GetLastError in several places because
// CloseInternetHandle clears the error code.
//
DWORD err;
HINTERNET h = InternetOpen(...);
if (h!=NULL) {
   HINTERNET hftp = InternetConnect(..,INTERNET_SERVICE_FTP,..);
   if (hftp!=NULL) {
      HINTERNET hftpfile = FtpOpenFile(...);
      if (hftpfile!=NULL) {
         InternetReadFile(...);
         if (/* success */) {
            // do something
         } else {
            err = GetLastError();
         }
         InternetCloseHandle(hftpfile);
      } else {
         err = GetLastError();
      }
      InternetCloseHandle(hftp);
   } else {
      err = GetLastError();
   }
   InternetCloseHandle(h);
} else {
   err = GetLastError();
}
Figure 4 InternetHandle.h (Close Figure 4)
////////////////////////////////////////////////////////////////
// MSDN Magazine — April 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#pragma once

//////////////////
// This handy class encapsulates an Internet handle so you never have to
// remember to call CloseInternetHandle. More importantly,
// CInternetHandle lets you postpone the call to CloseInternetHandle
// until control leaves the scope of your function, so you can get error
// information when something bad happens before CloseInternetHandle
// clears the error code.
//
class CInternetHandle {
protected:
   HINTERNET m_handle; // underlying handle

public:
   CInternetHandle() : m_handle(NULL) { }
   CInternetHandle(HINTERNET h) : m_handle(h) { }

   ~CInternetHandle() { Close(); }

   // Close handle and set to NULL so I don't close again.
   void Close() {
      if (m_handle) {
         InternetCloseHandle(m_handle);
         m_handle = NULL;
      }
   }

   // Assignment from HINTERNET
   CInternetHandle& operator= (HINTERNET h)
   {
      ASSERT(m_handle==NULL); // only initial assignment, not copy
      m_handle = h;
      return *this;
   }

   // cast to HINTERNET
   operator HINTERNET() {
      return m_handle;
   }
};

As MFC programmers know, MFC already has classes that encapsulate WinInet, so why didn't I use MFC instead of writing my own CInternetHandle? Because it's a waste to load all of MFC when all I need is a few lines of code. Less code is usually better than more. By making CWebVersion independent of MFC, non-MFC apps can use it too—without worrying whether users have the latest DLLs installed. That's a big plus, especially for applications you deliver over the Web.

Figure 7 Get Update Dialog
Figure 7 Get Update Dialog

To put everything together in a real app, I added over-the-Web version checking to the TraceWin program from last month's column. Figure 6 shows the changes (you can download the full program from the link at the top of this article). TraceWin reads the Web version when it starts, and stores it in a data member m_dwNewVersion (TraceWin only uses 32 bits for its version number). The About dialog compares this version number with the one stored in the program's VS_VERSION_INFO resource, which was stored when the program was compiled. If the Web version is newer, the dialog displays a "Get version XXX" link (see Figure 7). If the Web version is the same or unavailable (because the user is offline), TraceWin hides the link. Please note that TraceWin reads the version number only once, when it first starts—not every time the user invokes About. Reading a 10-byte file over FTP is fast enough to escape notice during startup, but slow enough to delay a dialog. Even with small files, you should be careful when and where you FTP. If you want the fastest possible UI, download the file in a separate thread. Happy programming!

Figure 6 TraceWin.cpp (Close Figure 6)
////////////////////////////////////////////////////////////////
// Changes to TraceWin to add live version checking.
//

#include "stdafx.h"
#include "WebVersion.h"

/////////////////
// Application class
//
class CTraceWinApp : public CWinApp
{
public:
   DWORD m_dwNewVersion;
   •••
} App;

BOOL CTraceWinApp::InitInstance()
{

   ••• // create frame, etc

   // read current version number from Web
   m_dwNewVersion = 0; // assume failure
   if (CWebVersion::Online()) {
      CWebVersion ver(_T("ftp.dilascia.com"));
      if (ver.ReadVersion(_T("TraceWinVer")))
         m_dwNewVersion = ver.dwVersionMS;
   }

   return TRUE;
}

//////////////////
// Custom about dialog uses CStaticLink for hyperlinks.
//
class CAboutDialog : public CDialog {
protected:
   •••
   CStaticLink m_wndLink3; // add hyperlink to download update..
   CStatic m_wndVersion;   // ..and static control to display my version
};

BOOL CAboutDialog::OnInitDialog()
{
   •••
   // subclass static controls
   m_wndLink3.SubclassDlgItem(IDC_URLUPDATE,this);
   m_wndVersion.SubclassDlgItem(IDC_VERSION,this);

   // Read my own version info and set static text
   CMyVersionInfo vi;
   CString s;
   s.Format(_T("TraceWin Version %d.%02d by Paul DiLascia"),
      HIWORD(vi.dwProductVersionMS),LOWORD(vi.dwProductVersionMS));
   m_wndVersion.SetWindowText(s);

   // compare my version to current version on Web site
   if (App.m_dwNewVersion && vi.dwProductVersionMS < App.m_dwNewVersion) {
      // current version is newer: display link
      s.Format(_T("Get version %d.%02d"),
         HIWORD(App.m_dwNewVersion),LOWORD(App.m_dwNewVersion));
      m_wndLink3.SetWindowText(s);

   } else {
      // current version not available: hide link
      m_wndLink3.ShowWindow(SW_HIDE);
   }
   return TRUE;
}

//////////////////
// Handle Help | About : run the About dialog
//
void CTraceWinApp::OnAppAbout()
{
   static CAboutDialog dlg;
   dlg.DoModal();
}

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLasciais a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at askpd@pobox.com or http://www.dilascia.com.


Resource File: ASP.NET Starter Kits

Resource File
ASP.NET Starter Kits

On this page we provide information on the most current Microsoft developer resources. This month we introduce you to the new ASP.NET Starter Kits, valuable tools for anyone creating Web sites with ASP.NET.

It's a well-known axiom in the world of programming: build the samples and they will come. ASP.NET is well suited for building online storefronts and communities, but a lack of easy-to-use, standardized samples has sometimes slowed adoption. To address this, Microsoft has just released a series of new ASP.NET application samples known as the ASP.NET Starter Kits. There are five individual kits designed to address specific needs for an organization, ranging in functionality from e-commerce storefronts to community portals. They include the Community Starter Kit, the Time Tracker Starter Kit, the Reports Starter Kit, the Commerce Starter Kit, and the Portal Starter Kit. Written in C#, Visual Basic .NET, and J#, the kits offer a set of common design patterns and provide valuable tools for developers creating ASP.NET Web sites.

The ASP.NET Community Starter Kit allows you to set up and configure a portal site, complete with discussion forums, photo albums, polling, content rating, and an automatically generated e-mail newsletter. User and role management features allow you to selectively grant access to content on your site. The Starter Kit makes integration with other sites easy by allowing you to selectively expose content as an XML-based Web Service which other sites can incorporate. The architecture of the Community Starter Kit was developed in such a way that is completely skinnable. Besides being a great example of how to separate code from content, this architecture makes it easy to customize the appearance of your site by creating your own Web skins.

The ASP.NET Time Tracker Starter Kit is built for any organization or team that works on a project basis. This Starter Kit allows team members to maintain a log of completed work as well as to log time spent on each project task (for example, on development or testing) through a Web interface. Using this data, project managers can then generate reports on the project, track overall progress, and analyze time allocations for individual projects.

ASP.NET Reports Starter Kit is a series of eight samples which demonstrate how to build various data reports using ASP.NET controls. The samples include master-details, visual, drill-down, cross-tab, hierarchical reports, and more. This kit allows you to create data-generated graphs, charts, sortable columns, and running totals.

The ASP.NET Commerce Starter Kit and the Portal Starter Kit are updated versions of the popular IBuySpy Store and Portal samples, respectively. Together they demonstrate how to build a Web storefront and simple portal application, including search functions, instant order services, shopping carts, and user account logon. The Commerce Kit offers everything you need to produce a catalog—room for product descriptions and photographs, ability to highlight popular items or "also bought" products.

The Portal Starter Kit helps you build a fully functional portal application with a dynamic infrastructure. It features 10 basic portal modules, role-based security, and mobile support using the ASP.NET mobile controls.

The ASP.NET Starter Kits are a valuable resource for both new and experienced ASP.NET developers. New developers can use Starter Kits to create Web sites without writing any code. Experienced developers can browse through the Starter Kit code and supporting whitepapers to understand the best practices for building ASP.NET applications.

Installation requirements include ASP.NET, SQL Server 2000 or Microsoft Data Engine version 7.0, and Windows 2000 or Windows XP. The ASP.NET Starter Kits are available for download at http://www.asp.net/starterkits.