 MSDN Magazine November 2007

MSDN Magazine November 2007

Trustworthy Computing: Lessons Learned from Five Years of Building More Secure Software

Five years ago, Bill Gates issued a directive to enhance security across the board. Since then, many valuable lessons have been learned about building more secure software.

Michael Howard Crash Course: Analyze Crashes to Find Security Vulnerabilities in Your Apps

Here the authors analyze program crashes to help you understand if you have the potential for read or write violations in your applications, and how they can lead to security vulnerabilities.

A. Abouchaev, D. Hasse, S. Lambert, and G. Wroblewski Code Reviews: Find and Fix Vulnerabilities Before Your Application Ships

Code defects can be found using many approaches, but manual code reviews stand out in terms of precision and quality. We provide some best practices for planning and executing code reviews on your own team.

M. Chmielewski, N. Clift, S. Fonrobert, and T. Ostwald Fuzz Testing: Create a Custom Test Interface Provider for Team System

Dan Griffin shows the extensibility of Visual Studio 2005 Team Edition for Software Testers by discussing the modification of the existing Test Interface Provider sample in the latest Visual Studio SDK and implements Fuzz Testing.

Dan Griffin Editor's Note: Why Go to Extremes?

This month Howard Dierking sheds some light on what we really mean when we talk about security.

Howard Dierking Toolbox: Updated grid controls, open-source comparison tools, and more

This time: Windows Forms grids, professional-looking date textboxes, file merging, and more.

Scott Mitchell CLR Inside Out: Managing Object Lifetime

Although the .NET Framework provides a managed execution environment, it is important to consider object lifetime management and write code that uses and releases resources correctly. The CLR team shows you how.

Tim Fischer Basic Instincts: Extension Methods

This column discusses one of the new features of Visual Basic 2008—Extension Methods. This powerful new feature was introduced to support LINQ. See what extension methods are, how to apply them yourself, and get some great usage tips.

Adrian Spotty Bowles Office Space: Events in SharePoint 2007

Windows SharePoint Services (WSS) 3.0 provides a new and improved infrastructure for handling server-side events. In this installment of Office Space, we look at techniques for hooking up Before Events and After Events using both Features and code.

Ted Pattison Cutting Edge: Managing the User Experience in AJAX

This month Dino takes a look at limitations and UI issues in Partial Rendering AJAX pages and techniques for managing the UI.

Dino Esposito Team System: Custom check-in policies

Team Foundation Server provides APIs that let you create check-in notes (categorized string data) and custom check-in policy implementations. These features combine to give project administrators better control over the group development process.

Brian A. Randell Foundations: Synchronization Contexts in WCF

After a description of what synchronization contexts are and how WCF uses them, the author demonstrates various options for extending WCF to use custom synchronization contexts, both programmatically and declaratively.

Juval Lowy Bugslayer: Measuring the Impact of View State

Bloated view state can be a real performance bottleneck for your Web app, but it can be difficult to diagnose. John Robbins creates a handy tool that records and reports the view state size for pages in your ASP.NET applications.

John Robbins Concurrent Affairs: Simplified APM with C#

Jeffrey Richter introduces his AsyncEnumerator class and explains how it harnesses some recent additions to the C# programming language that make working with the asynchronous programming model significantly easier.

Jeffrey Richter .NET Matters: Debugging finalizers

Find out how to use finalizers as a way to warn developers who use your custom types when they are garbage collected without having been disposed of correctly.

Stephen Toub { End Bracket }: WPF is for developers, too

There's still a role for developers in creating applications with WPF and XAML. In fact there are three: plumber, widget author, and ground breaker.

Kevin Moore


Trustworthy Computing: Lessons Learned from Five Years of Building More Secure Software

Trustworthy Computing

Lessons Learned from Five Years of Building More Secure Software

Michael Howard
This article discusses:
  • Prioritizing code by age
  • Using analysis tools and automation
  • Looking at threats from multiple angles
  • The importance of education
Contents
It's Not Just the Code
Fix Old Code First
Deprecate! Eliminate! Eradicate!
Tools Are Critical ... to a Point
Automate!
You'll Never Reach Zero Security Vulnerabilities
Security Is a Never-Ending Battle
There Is No Security Silver Bullet
The "Many Eyeballs" Mantra Is Right!
Today's Denial of Service Is Tomorrow's Exploit
Final Thoughts

F
ive years ago, Bill Gates issued a memo to all Microsoft employees explaining the importance of building more secure software. Since then, many people across Microsoft have worked to improve the security of their products. In doing so, we've learned a lot about what it takes to build more secure software.

Security is not a static field—it constantly evolves as attackers attack, defenders defend, and each party learns more about the other's techniques. Security is an arms race. To stay ahead and anticipate the attackers, we the defenders must learn from our mistakes and create better ways to secure users from being compromised.

So what have we learned in the past five years? Most of these lessons are actually quite obvious, but like so many apparent things, sometimes it helps to have someone point them out.

It's Not Just the Code

The software industry, or more accurately the software quality industry, is fixated on getting the code right. I really don't have a problem with that, but many security vulnerabilities are not coding issues at all. Many are design issues. If you focus solely on finding security issues in the code, you'll miss an entire class of vulnerabilities. This is one of the reasons Microsoft mandates threat modeling and attack surface analysis as part of the Security Development Lifecycle (SDL) Process. Threat Modeling is an analysis technique that helps identify and mitigate design weaknesses in a product. Attack surface analysis focuses on which portions of a software product are exposed to untrusted users, be they local or remote. A product with a large attack surface has more code exposed to untrusted users than a product with a small attack surface. (Read more at msdn.microsoft.com/msdnmag/issues/04/11/AttackSurface.)

The DNS RPC buffer overrun vulnerability documented in Microsoft Security Bulletin MS07-029 (see microsoft.com/technet/security/Bulletin/MS07-029.mspx) could allow an attacker to take complete control of an affected system. There was certainly a code issue in this case. However, the code was accessible to anonymous and remote users when it should really have been restricted to administrators. Thus, the problem was a combination of code and design vulnerability. I analyzed this vulnerability on the SDL blog (see blogs.msdn.com/sdl/archive/2007/06/28/lessons-learned-from-ms07-029-the-dns-rpc-interface-buffer-overrun.aspx).

Lesson Learned It's essential to build threat models to uncover potential design weaknesses and determine your software's attack surface. You need to make sure that all material threats are mitigated and that the attack surface is as small as possible.

Fix Old Code First

When it comes to code review, I like to rank code by potential for vulnerability. I wrote an article for IEEE Security & Privacy, titled "A Process for Performing Security Code Reviews," that explains the metrics I use to prioritize code review (you can find a link to the paper on my blog at blogs.msdn.com/michael_howard/archive/2006/08/01/686029.aspx).

The first priority is old code because old code is far more likely to have more security vulnerabilities than newer code. Threats are constantly evolving. Old code—even code built just a few years ago—was built when the threats were different than they are today. Furthermore, the techniques used to build old code lack the latest defensive techniques and best practices. Likewise, legacy code was built using older, less secure libraries. And finally, old code was built with less situational awareness, at a time when most developers had little to no security expertise.

My advice is simple: all old code must be hand-reviewed for security vulnerabilities. In fact, this is the purpose of the security push phase of the SDL here at Microsoft. While the primary goal of the SDL is to reduce the chance of developers adding new vulnerabilities to a product, the security push is designed to force the development team to look for issues in old code.

No other metric that we measure is as valuable when prioritizing code review—not code complexity, not line number count, not code churn. The number-one indicator of potential vulnerability density is simply the age of the code.

Lesson Learned Identify all your source code files and rank them by age, where age is the code's "born on" date. Perform static analysis and manually review the code, starting with the oldest code first.

Deprecate! Eliminate! Eradicate!

Sometimes a feature may not be secure enough in the current threat landscape. The feature may have been fine a few years ago, but it is unsecure today due not to code vulnerabilities but instead due to changes in today's computer environment.

A good example of this is the Alerter service in Windows® that would show print status and send little messages that would pop-up on another user's screen. This turned into a spam mechanism pretty quickly so we made the hard decision to disable the service by default in Windows XP SP2, and then we removed it entirely from Windows Vista®.

Another good example is the legacy IPX and SPX protocols. (Yes, I know that IPv4 is ancient too, and it has its own issues.) In Windows Vista, we simply removed support for the Microsoft® Client for NetWare Networks because the code was old and unused by most users.

Over time, you will see Microsoft carefully remove older features. Since some users rely upon features, it's important to balance the risk versus the usefulness.

Lesson Learned Identify old features that may be long-term security headaches and come up with a plan to deprecate the features. Perhaps the first step is to continue to ship the feature, but disable it by default. Then, in the next version of the product, the feature can be removed entirely, but made available as a Web download for those who absolutely depend on it. Finally, stop support for the feature. Just remember to keep your customers informed.

Tools Are Critical ... to a Point

In the past, I have been highly critical of tools. Actually, not of the tools themselves, but of the over-reliance some developers have on tools. By tools, I mean static code analysis, binary analysis, and the like that can help pinpoint security vulnerabilities. In my old age, I've softened somewhat on this opinion.

If you have a lot of code—say, over a million lines—it becomes very hard to review all that code by hand. Tools are handy because they can analyze great swaths of code rapidly. Tools, however, are no replacement for human intellect, and many tools tend to miss vulnerabilities because they are concerned with keeping the false positive and false negative rate as low as possible. And, to be totally honest, many security analysis tools create such a vast quantity of errors and warnings that it can be very hard to determine which bugs are real and which are noise.

Of course, if you do see a large number of issues, this doesn't mean you can simply ignore the tool output! When we perform a root cause analysis of a security vulnerability, we always ask why the issue wasn't discovered by our tools. There are three possible reasons: the tool did not find the vulnerability, the tool found it but mistakenly triaged the issue as low priority, and the tool did actually find the issue and humans mistriaged it. This analysis allows us to fine-tune our tools and education over time.

Analysis tools are also very good at determining the potential for security vulnerabilities in code. Say you have two products, each comprising about 100,000 lines of C++ code. You run your tools on each code base—for this example, assume the tools you use are /W4 and the /analyze compiler switches. The first code base yields 121 /W4 warnings and 19 /analyze warnings, while the second code base has 235 /W4 warnings and 65 /analyze warnings. Which set of code do you think needs more review?

Finally, tools are excellent when run on new or modified code prior to check-in, as they can act as code cops and find certain classes of bugs early in the process.

Lesson Learned Analysis tools can help you determine how much care you need to give your code. You can also use the output of the analysis to determine overall code riskiness. Use tools at check-in time to catch bugs early on. And run the tools often so you can deal with any new issues quickly—if you run the tools only every few months, you may end up having to deal with hundreds of warnings at a time.

Automate!

At the outset of a security improvement regimen, there is a great deal of manual work—manual code review, manual design reviews, and so on. To really elevate your work, you need to automate as much of the process as possible.

On our team, we built many bespoke tools and created an internal Web site to gather data from the tools so the central security team can review the output from tools. When an SDL improvement is proposed, the proposal must include a way to automate the improvement. The prime motivators for automation are scalability and constant use. If you have a great deal of code, you need to automate. And if you want parts of a process to be repeated constantly, then automation is obviously the way to go.

Lesson Learned Always strive for automation where possible. Build or buy tools that scan code and upload the results to a central site for analysis by security experts.

You'll Never Reach Zero Security Vulnerabilities

It's sad but true, but you'll never get to zero security vulnerabilities. I remember when we issued one of the first security updates for Windows Vista. Some users were surprised because they thought Microsoft claimed to have solved the security problem with Windows Vista. First, I don't know of anyone who made the claim and second, zero security vulnerabilities just isn't achievable.

While zero security vulnerabilities would be nice, thinking you can reach such a state is folly. The fact is the technology landscape is always in flux, threats are a moving target, and security research is ongoing. I said earlier security is an arms race. We add defenses to our products and the attackers adapt.

Your code might seem utterly vulnerability-free today, but that could all change tomorrow when a new type of vulnerability is discovered. For instance, on October 15, 2003, Microsoft issued a security bulletin that fixed a cross-site scripting (XSS) vulnerability in Outlook® Web Access included with Microsoft Exchange 5.5. On March 4th the following year, Sanctum (since purchased by Watchfire and now IBM) released a paper that outlined a new vulnerability akin to cross-site scripting called HTTP response splitting. Six months later, Microsoft issued another security update for Outlook Web Access in Microsoft Exchange 5.5 to fix an HTTP response splitting vulnerability. So what happened? Simply put, at the time the first bulletin was issued, response splitting issues were unheard of, but the landscape changed.

Lesson Learned Make sure people within the organization realize that your goal is to reduce security vulnerabilities, but that you will never reach zero security issues as long as there are still attackers looking for new techniques and vulnerabilities.

Security Is a Never-Ending Battle

There will never come a time when you can say, "we're done with security; the problem is now solved." This is an extension of the previous point, but with a slightly different angle. It's critically important that you maintain on-going training for your engineers. If you don't, skills might fade and urgency might dissipate over time. Security is critical and protecting systems is paramount. As I said in the previous point, new threats and vulnerabilities are constantly being discovered, so you must always treat security as a task that is never completed.

You should also realize that there is nothing special about security anymore. Security is simply part of getting the job done.

Lesson Learned Continue to provide on-going training for your engineers, and ensure that they are always aware of the importance of addressing security issues.

There Is No Security Silver Bullet

People often ask me "what is the most important element of the SDL?" The answer: everything. If a portion of the SDL proved not to be useful, it would be cut from the SDL. Every portion of SDL leads to a reduction in security vulnerabilities. So when you hear a person say he has a single, magical task to address security, such as running static analysis tools or educating people, he isn't covering all his security bases. Sure, static analysis tools have their place, and educating users is essential, but not on their own. Security must be thorough, and it must be part of the process.

Lesson Learned Make sure you are addressing security from every angle available. If not, you need to change your process!

The "Many Eyeballs" Mantra Is Right!

Famed open source advocate Eric Raymond has said, "given enough eyeballs, all bugs are shallow." He's right. But I think this simplified statement misses a key point. It should continue, "so long as the eyeballs are incented and educated."

At Microsoft, we can require people to adopt process improvements since these requirements are a part of the job. That's incentive. We offer education in many forms, such as live training, labs, and online training. Thus, there's plenty of material to keep engineers attune to what is happening on the security front.

Lesson Learned The more eyes on your code the better, but this must be done within a certain framework. You must ensure that the developers providing code review have an incentive to care about adapting to your current process, and they must be educated about the latest security threats.

Today's Denial of Service Is Tomorrow's Exploit

Many software vendors, including Microsoft, have learned this lesson the hard way. Engineers might look at a code issue and quickly dismiss it because the issue is "just a DoS." Remember that the security research world is constantly evolving and assumptions about certain classes of vulnerability can change overnight.

Lesson Learned Don't be quick to dismiss a denial of service issue. With a little work, malicious users can turn some DoS vulnerabilities into real code execution vulnerabilities.

Final Thoughts

If there is one thing I've learned in the past few years, it's that when it comes to security, you need to be prepared to have your ideas and viewpoint constantly challenged and you must be proactive about staying abreast of the latest issues. If you take the lessons learned in this article to heart, you should do just fine.

Michael Howard is a Principal Security Program Manager at Microsoft focusing on secure process improvement and best practices. He is the coauthor of many security books, including Writing Secure Code for Windows Vista, The Security Development Lifecycle, Writing Secure Code, and 19 Deadly Sins of Software Security.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Crash Course: Analyze Crashes to Find Security Vulnerabilities in Your Apps

Crash Course

Analyze Crashes to Find Security Vulnerabilities in Your Apps

Adel Abouchaev and Damian Hasse and Scott Lambert and Greg Wroblewski
This article discusses:
  • Causes for program crashes
  • Exploitable vs. non-exploitable errors
  • Analysis best practices
  • Strategies for triaging errors
This article uses the following technologies:
Windows debugging, C/C++
Contents
Access Violations
Access Violation When Writing Data
Non-Exploitable Exceptions
/GS Exceptions
NX Exceptions
Putting It into Practice

H
ow can you make sure a crash in your program is not exploitable? The short answer is simple: assume every crash is exploitable and just fix it! At the very least, it is a matter of quality and it is often cheaper—and more practical—to fix the issue before the product has been shipped to customers. It can be rather expensive to perform the analysis necessary to determine exploitability.

Analyzing program failures related to memory corruption in order to understand the security ramifications can be a complex and error-prone task. Several factors must be considered, including the location of the buffer in memory, the possible targets for overwriting, the size of the overwrite, restrictions on the data that can be used during the overwrite, the state of the runtime execution environment, and the ability to bypass any mitigation mechanisms in place. In a nutshell, you must understand the root cause of the failure in order to answer these questions thoroughly.

Keep in mind that not every failure will manifest itself in an observable manner. One example of this is the GDI remote code execution issue discussed in Microsoft Security Bulletin MS07-017. The software responsible for invoking the vulnerable parsing code made use of an exception handler to recover from pretty much any exception that could be generated and continue operating as if nothing had occurred. (Read more about it at blogs.msdn.com/sdl/archive/2007/04/26/lessons-learned-from-the-animated-cursor-security-bug.aspx.) Another example that's less obvious can be found in certain types of stack and heap memory corruptions, where it's possible that failures have occurred but the current state of the program and its execution environment don't show any obvious signs.

This article offers guidance on how to analyze program crashes with regard to possible security implications—such as the case of memory corruption enabling arbitrary code execution or at the very least denial of service. We will enumerate the common hardware and software exceptions you might encounter when looking at these types of issues. We'll also offer some general guidelines you can use during such an investigation. Figure 1, for example, gives you a graphical path of the investigation process to assist you in deciding whether a particular crash is exploitable. It is important to remember that these are just guidelines and that only a full root-cause analysis can ensure that you have correctly diagnosed the crash as not being exploitable. New techniques or variations of existing attack techniques are being discovered all the time.

Figure 1 Access Violation Analysis Path
Figure 1 Access Violation Analysis Path(Click the image for a smaller view)
Figure 1 Access Violation Analysis Path
Figure 1 Access Violation Analysis Path (Click the image for a larger view)

The most common cause of crashes is hardware or software exceptions. A typical modern processor can generate many different types of hardware exceptions, but in the Windows® environment only some of those create problems related to software security. The most common hardware exception is an access violation. We will first cover how to analyze hardware exceptions, followed by software ones.

Access Violations

Access violation exceptions (0xc0000005= STATUS_ACCESS_VIOLATION) are generated by modern processors when a memory access caused by an instruction or program execution does not satisfy certain conditions defined by the processor architecture or memory management unit structures.

While a pure crash can only result in a denial of service condition, it is not safe to assume that a crash can't be used to achieve more dangerous effects, including code execution. In analyzing crashes, you should assume that the entire body of memory, with some minor exceptions, is under control of a potential attacker; thus an access violation can in most cases lead to data controlled by the attacker. This statement applies to an exception that occurred while an instruction was either reading or writing data.

If the access violation can lead to your data being controlled by the attacker, then each access violation caused by a read from memory turns into loading an attacker's controlled data. The security effects of such an operation are not always easy to determine. You could perform a full data flow analysis on either the binary or the source code to find the scope of source address control and the consequences of feeding the program with random data at certain points of execution. This is a time-consuming and challenging task. In response, we have developed simple heuristics to quickly analyze read-access violation crashes for code-execution potential.

As the following example shows, an invalid memory pointer in the register eax caused a crash. In this case, control of the memory contents gives the attacker full control over program flow:

Application!Function+0x133:
3036a384 eb32            call     [eax]           ds:0023:6c7d890d=??
0:000> ub
mov    eax, [ebx]  ->  eax = invalid memory pointer
...                         (instructions not affecting register eax)
call        [eax]  ->  crash

Situations in which the address being read can't be sufficiently controlled by the attacker can be treated as denial of service conditions. For example, in the typical Windows user mode environment, a crash that occurs at a NULL pointer that was initialized and can't be affected by an attacker can't lead to code execution by itself.

In the following example, you can see that a crash was caused by referencing address zero through value in register eax:

Application!Function+0x133:
3036a384 8b14            mov     ecx, [eax]           ds:0023:00000000=??
0:000> ub
xor    eax, eax
...                         (instructions not affecting register eax)
cmp    ebx, 2
jne        label123
mov    ecx, [eax]  ->  crash, eax = 0 (NULL)

By disassembling (with the ub command in the Visual Studio® command-line debugger), we can track the data flow in this register until we confirm that the value in the register can't be influenced by malicious input. Indeed, in this example, the register was zeroed by XORing with itself and it wasn't used until the crashing instruction was reached.

Sometimes the exploitability of a defect is not immediately visible from the crashing instruction. For example, after disassembling the following instructions, you can see that it is a consequence of the control flow when the failing instruction (described in the previous paragraph) is followed by a critical one:

(1258.1638): Access violation - code c0000005 (second chance)

Application!Function+0x123:
3036a384 8b12            mov    eax, [ebx]           ds:0023:6c7d890d=??
0:000> u
mov    eax, [ebx]  -> crash, ebx points to invalid memory
...             (instructions not affecting regist an example er eax)
call        [eax]  ->  possibility of code execution

This example is similar to the first one, yet this time the crash occurred on the instruction loading data into the register eax. Although this operation by itself does not indicate security problems, the disassembly clearly shows that control over the value of register ebx implies control over the value of register eax, which leads to possible code execution.

The remaining cases can be analyzed by simulating malicious data injection during program run time, either manually under a debugger or automatically through a debugging tool. After reaching the instruction that causes the access violation, we can either change the source address to point to a valid memory address and execute the instruction again or we can put random data into the destination register and skip the faulting instruction, continuing execution. We will repeat this process for each access violation encountered until we reach an exploitable condition.

Let's analyze two examples. First, let's look at tracking data flow when we change the source address:

Application!Function+0xa70:
3036a37e 8b4708          mov     eax,dword ptr [edi+8] ds:0023:040fd004=????????

The crash here is caused by an incorrect value in register edi. We will change it to point to a valid memory region. There are many possible choices, but in practice we often use the current value of register eip. This ensures that a relatively large chunk of memory around the new value of edi is valid and adds the possibility of catching any subsequent writes to the memory block containing code, as it is always marked as read-only:

0:000> r edi=eip
0:000> g

After setting register edi to the current value of register eip, we continue execution, hitting another exception on register esi where we repeat the process:

(1258.1638): Access violation - code c0000005 (second chance)

Application!Function+0xa76:
3036a384 f60601          test    byte ptr [esi],1           ds:0023:6c7d890d=??
0:000> r esi=eip
0:000> g

Continuing execution, we hit a write access violation exception on an instruction trying to write data to the code segment. This means an attacker can also write data to any memory address:

 (1258.1638): Access violation - code c0000005 (second chance)

Application!Function+0xbef:
3036a4fd 894710          mov     dword ptr [edi+10h],eax ds:0023:3036a38e=0c46f60d

As you can see, the initial read access violation turned out to be a real security issue with possibilities for code execution.

Now let's look at tracking data flow by setting destination data and skipping instructions. Analyzing the same crash as before, we will keep changing the destination data. After the initial exception is hit, we set the eax register to any easy-to-track value and increase register eip by the size of the current instruction to skip over it:

Application!Function+0xa70:
3036a37e 8b4708          mov     eax,dword ptr [edi+8] ds:0023:03f93004=????????
0:000> r eax=deadbeef
0:000> r eip=eip+3
0:000> g

Continuing execution, we hit the next exception, which does not have a destination, so we can just skip the instruction:

(1258.1638): Access violation - code c0000005 (second chance)

Application!Function+0xa76:
3036a384 f60601          test    byte ptr [esi],1           ds:0023:deadbefb=??
0:000> r eip=eip+3
0:000> g

Executing further, we reach the same write access violation exception, implying the possibility of exploitable issue:

(1258.1638): Access violation - code c0000005 (second chance)

Application!Function+0xbef:
3036a4fd 894710          mov     dword ptr [edi+10h],eax ds:0023:03f9300c=0c46f60d

While both methods of simulating malicious data injection often yield the same results, from practice we can say that in many cases they cover different code paths, with only one method showing exploitability of an issue. All of the methods discussed so far allow finding potentially exploitable read access violation crashes very quickly, but we must remember that they do not provide a final validation that a certain crash is not an exploitable issue.

Access Violation When Writing Data
Crash Analysis Resources
  • MSDN Security Developer Center
  • Latest Microsoft Security Bulletins
  • Visual Studio Debugger Security
  • Debugging Tools for Windows

An access violation during data writing indicates possible memory corruption, which almost always leads to an exploitable condition with potential for code execution. Very often such writes are indicators of buffer overflow conditions present in the crashing program. In practice, a few write access violation crashes can be shown to be not exploitable. Nonetheless, these situations require you to perform a full data flow analysis to understand the root cause of the issue. You need to ensure that the corruption doesn't lead to an attacker being able to overwrite data to arbitrarily influence the flow of execution.

Most access violations with data writes can lead to exploitable cases that will result in malicious code execution. Typically code execution is achieved by overwriting an (arbitrary) piece of memory, either on the stack or on the heap.

In the following example, the crash occurred on a memory copying instruction when the destination register reached the upper boundary of the stack and hit an unallocated area of memory. The size of the copying operation came from the variable at address [ebp-8] and the crash indicates that it is under the attacker's control:

Application!Function+0x143:
3036a384 f3a4            rep     movsb           es:0023:00140000=??   ds:0023:0125432c=41414141
0:000> ub
mov    esi, [ebp-4]
mov    edi, [ebp+4]
mov    ecx, [ebp-8]
rep    movsb  ->  write access violation if value in ecx is big enough

We can say with confidence that a write access violation can't lead to code execution only if the destination addresses controlled by the attacker point to invalid memory or to data that has no influence on program execution. In practice, only a small set of write access violations satisfy this condition. In the Windows user mode environment, a write to a NULL pointer or system address space (usually addresses above 2GB) could be an example of a non-exploitable issue (assuming that the page with address 0x00000000 can't be allocated). Also, in a server scenario, the instruction would lead to a denial of service, which should be considered a security vulnerability if it is triggered by a non-administrator user.

This example is similar to the earlier case discussed regarding read access violations. Quickly disassembling shows that the register eax has been initialized to value zero and has not been changed until the crashing instruction was reached:

Application!Function+0x133:
3036a384 8d14            mov     [eax], ecx           ds:0023:00000000=??
0:000> ub
xor    eax, eax
...                       (instructions not affecting register eax)
cmp    ebx, 2
jne    label123
mov    [eax], ecx  ->  crash, eax = 0 (NULL)

Non-Exploitable Exceptions

In user mode—not in kernel mode—this class of exceptions is unlikely to be used in a single-stage exploit. Usually these exceptions result in the denial of service. Only in some cases, when the exception handler is changed as a result of another vulnerability, can these exceptions lead to malicious code execution. Denial of service is a high-priority bug in the server platform and medium priority in workstations.

An example of these exceptions is static/global dereferencing (read and write). The process in this example does not have read access to page 0x310000 for the read access violation to occur, or write access to the same page to trigger a write access violation.

mov ebp, 310046h
mov eax, [ebp+4h]

inc eax
mov [ebp+8h], eax

The first two exceptions need to be carefully analyzed. If the static/global value points to the address in memory where malicious code can write without limits in length and can overwrite other structures to stage the complex exploit, this case has to be marked as exploitable. If it only allows you to store a single DWORD on the address that is not part of any control structures, it is most likely not exploitable. However, this statement must be verified by running the program and observing the values (also known as runtime analysis). If the value that is stored is not used later in the code, it cannot lead to another exploitable condition and can be ignored. If the value could be used as a memory address or in a memory copy operation, the risk of exploitation is higher.

Code execution on a static/global address in uncontrollable address space (page 0 and similar cases) is another instance where exploitability relies on whether the memory page at the address from the operand is controllable for write:

0040137F B8 DE C0 AD DE    mov     eax, 0DEADC0DEh
00401384 FF D0             call    eax

Analysis of code execution on a static/global address has further implications. If the address belongs to the pages that would never be associated with normal process execution (page 0 or pages with addresses above 0x80000000), then it is not exploitable.

The division-by-zero operation will also trigger an exception, but it is not directly exploitable at this point. This situation needs extra analysis to decide whether the result of this exception can set the CPU to execute code that can lead to a successful exploit:

004013D6 33 C9    xor    ecx, ecx
004013D8 8B C1    mov    eax, ecx
004013DA 40       inc    eax
004013DB F7 F1    div    ecx

Unhandled C++ exceptions can disrupt the execution of the process. During runtime, they will cause termination of the application. Under the debugger, it is possible to continue after the unhandled exception. C++ exceptions occur when the runtime library's exception raise functions are called:

00401902    mov     [ebp+var_4], 1
00401909    push    offset __TI1H
0040190E    lea     eax, [ebp+var_4]
00401911    push    eax
00401912    call    __CxxThrowException@8 ; _CxxThrowException(x,x)

This condition can only be exploitable if the exception handler mechanism is already overwritten. Otherwise unhandled C++ exceptions are not exploitable. The stack trace for this exception is shown in Figure 2.

Figure 2 Stack Trace for an Unhandled C++ Error (Close Figure 2)
CommandLine: test.exe
Symbol search path is: srv*c:\Symbols*\\symbols\symbols
Executable search path is:
ModLoad: 00400000 00405000   test.exe
ModLoad: 7c900000 7c9b0000   ntdll.dll
ModLoad: 7c800000 7c8f5000   C:\WINDOWS\system32\kernel32.dll
ModLoad: 78130000 781cb000   C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_6b128700\MSVCR
80.dll
ModLoad: 77c10000 77c68000   C:\WINDOWS\system32\msvcrt.dll
(1494.14c4): Break instruction exception - code 80000003 (first chance)
eax=00251eb4 ebx=7ffda000 ecx=00000004 edx=00000010 esi=00251f48 edi=00251eb4
eip=7c901230 esp=0012fb20 ebp=0012fc94 iopl=0    nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
ntdll!DbgBreakPoint:
7c901230 cc              int     3
0:000> g
1
(1494.14c4): C++ EH exception - code e06d7363 (first chance)
(1494.14c4): C++ EH exception - code e06d7363 (!!! second chance !!!)
eax=0012fee0 ebx=00000000 ecx=00000000 edx=781c3c58 esi=0012ff68 edi=004033a4
eip=7c812a5b esp=0012fedc ebp=0012ff30 iopl=0    nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000206
kernel32!RaiseException+0x53:
7c812a5b 5e              pop     esi
0:000> kb
ChildEBP RetAddr  Args to Child
0012ff30 78158e69 e06d7363 00000001 00000003 kernel32!RaiseException+0x53
0012ff68 00401917 0012ff78 004022b8 00000001 MSVCR80!_CxxThrowException+0x46
0012ff7c 004011b2 00000001 00353098 003530d0 test!wmain+0x27
0012ffc0 7c816fd7 00011970 7c9118f1 7ffda000 test!__tmainCRTStartup+0x10f
0012fff0 00000000 004012fb 00000000 78746341 kernel32!BaseProcessStart+0x23

Note that the stack overflow exception may hide other issues at runtime, divert code flow to a different path, disable compiler generated protection mechanisms such as /GS, or bring inconsistency into the application by freeing memory that is still in use. When stack space is almost consumed, it is possible that an application will fail inside of C++ methods with a stack overflow exception and important pieces of code will not be executed in order. Additional triaging will be required in such conditions.

/GS Exceptions

/GS (0xc0000409=STATUS_STACK_BUFFER_OVERRUN) exceptions are those Windows will throw whenever it detects that the security cookie protecting the return address has been tampered with. Since the goal of /GS is to turn buffer overruns that lead to code execution into denial of service attacks, whenever such a crash is detected you can be certain you have a security bug. (Unfortunately, due to buggy memory, overclocked motherboards, faulty hardware, and other issues, sometimes the code that validates the cookie gets tripped without having an actual buffer overrun.)

In Windows Vista®, the operating system will throw an int 3 exception when a STATUS_STACK_BUFFER_OVERRUN is detected (assuming a debugger is present). In earlier versions of Windows, a breakpoint should be placed in kernel32!UnhandledExceptionFilter to detect whether a security cookie got tampered with (otherwise the process will get terminated and the user will not get notified).

In Figure 3, the function foo will overrun a buffer by copying too much data to a stack buffer, which will cause the /GS cookie to get overwritten.

Figure 3 Overwriting the /GS Cookie (Close Figure 3)
<the /GS cookie is being setup in the function prolog>
0:000:x86> u gs!foo
gs!foo:
010011b9 8bff            mov     edi,edi
010011bb 55              push    ebp
010011bc 8bec            mov     ebp,esp
010011be 83ec18          sub     esp,18h
<global cookie will be moved to eax register>
010011c1 a100200001      mov     eax,dword ptr [gs!__security_cookie (01002000)]
010011c6 53              push    ebx
010011c7 56              push    esi
010011c8 57              push    edi
010011c9 8b7d08          mov     edi,dword ptr [ebp+8]
<cookie will be placed on the stack (ebp-4)>
010011cc 8945fc          mov     dword ptr [ebp-4],eax
<content of the source (src) and destination buffers (dst) – before the overrun>
0:000:x86> dv /V
000bfefc @ebp+0x08             src = 0x009d16cb "123456789012345678901234567890"
000bfedc @ebp-0x18             dst = char [20] ""
<value of the security cookie on the stack; note that it is located right after the buffer, before saved ebp (0x000bff24) and the return address (0x0100124a)>
0:000:x86> dd 000bfedc+0n20 l3
000bfef0  0000b96f 000bff24 0100124a
<value of global cookie>
0:000:x86> dd gs!__security_cookie l1
01002000  0000b96f
... code runs ...
<after the overrun has happened, the cookie got overwritten (with 0x34333231) as well as the last two bytes of the return address (with (0x3039)>
0:000:x86> dd 000bfedc+0n20 l3
000bfef0  34333231 38373635 01003039
<in the function epilog before it returns the /GS cookie gets checked (i.e. the return address has not been used yet)>
0:000:x86> u gs!foo+0x54
gs!foo+0x54:
0100120d 59              pop     ecx
<cookie is placed in the ecx register>
0100120e 8b4dfc          mov     ecx,dword ptr [ebp-4]
01001211 5f              pop     edi
01001212 5e              pop     esi
01001213 5b              pop     ebx
01001214 e882020000      call    gs!__security_check_cookie (01002000)
01001219 c9              leave
0100121a c20400          ret     4
<the below functions compares the global cookie with the one that was stored in the stack (currently in ecx register)>
0:000:x86> r
eax=0000000c ebx=7efde000 ecx=34333231 edx=00000000 esi=00000000 edi=00000000
eip=0100149b esp=000bfed8 ebp=000bfef4 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
gs!__security_check_cookie:
0100149b 3b0d00200001    cmp     ecx,dword ptr [gs!__security_cookie (01002000)] ds:002b:01002000=0000b96f
<if a debugger was attached to the process, then an (int 3) will be issued (in Windows Vista) before the process gets terminated>
0:000:x86>
STATUS_STACK_BUFFER_OVERRUN encountered
 (15d0.1708): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
773e0004 cc              int     3

NX Exceptions

NX (0xc0000005=STATUS_ACCESS_VIOLATION) exceptions are thrown by Windows whenever it detects code executing on a page that is not marked executable (in other words, the page does not have PAGE_EXECUTE, PAGE_EXECUTE_READ, or other relevant flags). NX is enforced in 64-bit versions of the operating system. Some applications like unpackers and digital rights management (DRM) rely on executing code on the heap, so not every NX exception should be considered a security vulnerability. It still makes sense to understand the root cause of the bug to ensure that it does not have security implications.

It's important to note that this type of exception uses the error code 0xc0000005, which is not specific to NX; any application that misbehaves (by, say, reading from unallocated memory) can throw this error. In order to understand whether the exception/error code is indeed NX related, we need to review the protection set on the page. If the page is not marked executable, we have an NX exception; otherwise we have some other type of problem. In Figure 4, for example, a first chance exception was hit, but the instruction seems to be valid and it references valid data.

Figure 4 Potential NX Exception (Close Figure 4)
(1424.a78): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
00000000`003b0020 2801            sub     byte ptr [rcx],al ds:00000000`7702e6aa=c3
0:000> r
rax=0000000000000001 rbx=000000000021fd10 rcx=000000007702e6aa
rdx=0000000000000000 rsi=000000000000000a rdi=0000000000000000
rip=00000000003b0020 rsp=000000000021fca0 rbp=00000000ff130000
 r8=000000000021fc98  r9=00000000ff130000 r10=0000000000000000
r11=0000000000000244 r12=00000000ff131728 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00000000`003b0020 2801            sub     byte ptr [rcx],al ds:00000000`7702e6aa=c3
<when we execute the actual instruction again, we hit a second chance access violation>
0:000> t
(1424.a78): Access violation - code c0000005 (!!! second chance !!!)
00000000`003b0020 2801            sub     byte ptr [rcx],al ds:00000000`7702e6aa=c3
<a closer look at the protection of the page, shows that it does not have PAGE_EXECUTE set, this explains why the code cannot be executed>
0:000> !address 00000000`003b0020
 ProcessParametrs 00000000003b25c0 in range 00000000003b0000 00000000003e0000
 Environment 00000000003b1370 in range 00000000003b0000 00000000003e0000
    00000000003b0000 : 00000000003b0000 - 0000000000030000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageHeap
                    Handle   00000000003b0000

Putting It into Practice

As you can see, analyzing program failures for security implications is an involved and potentially error-prone task. We discussed the most common exceptions you are likely to encounter when analyzing crashes, including read/write access violations, division by zero, C++ exceptions, /GS exceptions, and NX-related issues. The key point to remember is that only a full root-cause analysis can ensure that you have correctly diagnosed whether a given crash is exploitable.

In an effort to provide you with some useful guidelines for analyzing your own apps, we have summarized the information into a quick reference you can keep by your workstation. Figure 5 captures the guidelines discussed in the article in table form. For example, if a Write Access Violation occurred as a result of the CPU trying to write data to a memory page without having write access to it, you need to fix the issue.

Figure 5 Triage Fix Criteria (Close Figure 5)

Exception | Comments
Must Fix |
Write access violation | The access violation happens when the CPU tries to write data to the memory page without write access to it.
Read access violation on the instruction pointer (access violation on EIP) | The access violation happens when the CPU tries to execute an instruction on the memory page without read access to it.
Read access violation | One of the following may occur: •The access violation happens on a rep assembly instruction (on an Intel processor) where the count register (ecx) is large. •The access violation happens on a mov instruction where the result is used as the destination of a call in the instructions immediately after the mov. •The access violation happens on a mov instruction where the result is later used in a rep instruction as the source (esi), destination (edi), or count (ecx).
Triage Required |
Read access violation | If the access violation happens reading from NULL (address 0x00000000) or if the access violation happens reading from the memory address that is not controlled by the input and the value is not manageable by the attacker.
Usually Not Exploitable |
Division by zero | If the access violation happens as a standalone issue and other structures (exception handlers, for example) are not corrupted before this access violation.
C++ exception | Same as the above.

Likewise, Figure 1 (introduced earlier in this article) is intended to provide another view of the process by using a graph structure to help you decide whether a particular crash is exploitable. You start with the top node and ask yourself which node to visit next. You continue this process until you arrive at either the Not Exploitable or Exploitable node.

We hope you find these guidelines useful when triaging program failures. To find more information, follow the links in the "Crash Analysis Resources" sidebar. And remember, if you diagnose a crash as exploitable in a Microsoft product, please send this information to secure@microsoft.com.

Adel Abouchaev (CCIE#12037, MCSE, CISSP) is a Security Software Engineer on the Secure Windows Initiative (SWI) team at Microsoft. He is responsible for fuzzing and runtime analysis tools and works to provide product teams at Microsoft with automated methods of security analysis and security evaluation.

Damian Hasse Lead Security Software Engineer at Microsoft, leads a team of security researchers that investigate vulnerabilities and security threats as part of the Microsoft Security Response Center (MSRC).

Scott Lambert is a Security Program Manager on the Secure Windows Initiative (SWI) team at Microsoft. He is responsible for the enhancement of internal security tools, including various fuzzing tools. Leveraging his industry experience, Lambert works to ensure that SWI tools identify the vast majority of vulnerability classes.

Greg Wroblewski is a Security Software Engineer at Microsoft working in a security response team.


Code Reviews: Find and Fix Vulnerabilities Before Your Application Ships

Code Reviews

Find and Fix Vulnerabilities Before Your Application Ships

Michal Chmielewski and Neill Clift and Sergiusz Fonrobert and Tomasz Ostwald
This article discusses:
  • The code review process
  • Prioritizing your code for review
  • Types of security vulnerabilities
  • Using the review findings
Contents
Software Security Vulnerabilities
Finding Coding Errors
The Code Review Process
Prioritizing Review Efforts
Code Reviewing Tactics
Types of Vulnerabilities
Processing Results
Wrapping Up

I
n software development, a small coding error can result in a critical vulnerability that ends up compromising the security of an entire system or network. Many times, a security vulnerability is not caused by a single error, however, but rather by a sequence of errors that occur during the course of the development cycle: a coding error is introduced, it goes undetected during the testing phases, and available defense mechanisms do not stop a successful attack.

Security must be a priority in all phases of software development. Effort should be aimed at preventing software vulnerabilities—detecting them before release, of course, but also limiting their practical impact (for example, by reducing the product's attack surface). At Microsoft, such a holistic approach to security is implemented through the Security Development Lifecycle (SDL), which covers all major phases of software development, including educating developers, improving design, employing coding and testing practices, and preparing for emergency responses after the release of a product, as you can see in Figure 1. SDL is not the only way to approach code review, but it does form the basis for much of what we'll cover here.

Figure 1 The Security Development Lifecycle
Figure 1 The Security Development Lifecycle(Click the image for a smaller view)
Figure 1 The Security Development Lifecycle
Figure 1 The Security Development Lifecycle (Click the image for a larger view)

In this article we will discuss manual security code reviews performed by developers or security experts. In a process defined by the SDL, such efforts usually take place during a security push or penetration-testing engagement and are associated with a final security review. Coding errors can be found using different approaches, but even when compared to sophisticated tools, manual code reviews have clearly proven their value in the areas of precision and quality. Unfortunately, manual code reviews are also the most expensive to execute.

We also intend to discuss in detail the advantages and disadvantages of security code reviews in the context of large software projects. This article has been prepared based on experiences gathered over time, through reviews of major products released by Microsoft over the last few years.

Software Security Vulnerabilities

A software product of nontrivial size and complexity should never be assumed free of security vulnerabilities. All possible steps should be taken to limit the number of coding errors and reduce their practical impact, but something is always missed. Software errors that affect security (referred to as vulnerabilities) can exist at different levels of the application and be introduced during different phases of the development cycle.

Vulnerabilities are not limited to code. They can be introduced as early as the requirements definition in the form of a requirement that cannot be implemented in a secure manner. The basic design of a product may also contain flaws. For example, inappropriate technologies may be selected or their use may be incorrect from a security point of view. Ideally all of these problems will be identified by design reviews or threat modeling during the early stages of product development.

Security code reviews are primarily aimed at finding the code-level problems that still cause a majority of security vulnerabilities. These efforts may result in the identification of design issues as well, but such issues could be related to needed improvements in threat modeling and other aspects of the development process. Source code reviews can also be conducted with non-security-related priorities. But in this specific context, the goal is to find code vulnerabilities that could be used to either break security guarantees made by the product or to compromise the security of the system.

A coding error that can potentially cause security problems (such as problems due to lack of validation) must fulfill specific conditions to constitute a security vulnerability. There must be a security boundary to attack, and an attacker needs to have some level of control over the data or the environment. Problems that may exist in code that is executed within the same security context as the attacker offer no potential privilege gain in exploiting the vulnerability. In other cases, vulnerabilities exist, but they are located in code that cannot be executed because it is not accessible by the attacker. These coding errors, although they may affect product reliability, should not be considered actual vulnerabilities. The ultimate goal of security code reviews is to find code vulnerabilities that are accessible by an attacker and that may allow the attacker to bypass a security boundary.

Note that the accessibility of a vulnerability is not equivalent to its exploitability; a successful attack may still be mitigated by platform enhancements such as the /GS flag, the /SafeSEH flag, or Address Space Layout Randomization (ASLR).

The exploitability of a code vulnerability is not in the scope of investigations performed during code reviews, primarily because it is usually impossible to prove that available exploitation mitigations are sufficient. The responsibilities of a code reviewer end with confirmation that a code vulnerability exists and that it is appropriately triaged. From that moment the bug should be considered simply another problem that requires a fix.

Identifying code vulnerabilities is a primary goal of security code reviews, but there can be additional outcomes from the effort. A reviewer may provide feedback about overall code quality, redundancy, dead code, or unnecessary complexity. Reviewers may also deliver recommendations for improvements in reducing the attack surface, data validation, code clean-up, or code readability (such as improving comments). However, since documenting such outcomes takes time, you should decide before starting whether results should also include such recommendations or whether efforts should focus solely on identifying security problems.

Finding Coding Errors

There are different approaches to finding code errors. Since each has both unique advantages and practical limitations, it is important to understand the difference between code reviews and other options. For the purposes of this article, we will assume that both source code and design documentation are available, and that a so-called "white-box" analysis of the product is performed (an internal analysis, as opposed to a "black-box" approach that focuses only on externally visible behavior).

Code review can be described as a manual and static approach to finding coding errors. Using similar descriptions, two other popular approaches can be described as an automated static approach and an automated dynamic approach. The automated static approach usually takes the form of static code analysis tools that operate on source code in an attempt to identify known types of problems defined using patterns. PREfix and PREfast are examples of this approach. The automated dynamic approach is represented by automated code testing techniques (such as fuzzing), which are primarily focused on files, protocols, and APIs. Although these solutions can be applied also in a black-box approach, much better results can often be achieved if information about internal elements such as file formats is available and used appropriately.

Each approach has certain practical advantages and limitations. Static code analysis tools allow more code to be processed through automation, but findings are strictly limited by the set of predefined patterns for known types of problems. The results may often also contain a large number of false positives that make addressing issues difficult with limited resources. Fuzz testing can be easily automated and conducted on a continuous basis, but it operates in at least a partially random manner and may have problems with reaching deeper parts of the code. In most cases it is relatively easy to conduct basic fuzzing, yet it is much more difficult to achieve complete coverage of critical code paths.

Compared to automated methods, the advantages and disadvantages of manual source code reviews are connected primarily with direct involvement of a human reviewer. Results of efforts may differ significantly because they depend on a participant's experience with specific technologies, architectures, and scenarios. Generally, it is wise to assume that a human reviewer still has advantages over a machine-based solution. The reviewer can learn throughout the process, understand the context of software components, and interact with designers and developers. Additionally, a reviewer can provide feedback that is not limited only to a detailed issue report, but can also include high-level recommendations.

At the same time, however, a reviewer is susceptible to human error, fatigue, or boredom. The scope of a review is usually limited, and evaluating the quality of the review may be difficult since it is usually based on the subjective confidence of the reviewer.

These disadvantages can be overcome, however. One can design a task-specific review, aimed at analysis of a code block in multiple passes for only one type of error at a time. Using another approach, multiple reviewers can perform independent reviews of a critical piece of code, thus limiting the probability of human error. Code reviews are one of the specific cases where redundancy has huge potential value as it allows overcoming the limitations of human involvement.

Manual code review should never be considered as the ultimate solution for finding code vulnerabilities or as a replacement for other approaches, but rather as a complementary solution. Code reviewing should never be introduced to replace threat modeling, fuzzing, or enforcing coding best practices. Compared to other approaches, code reviewing is usually more expensive, so it should be used primarily in the scope of the most critical areas and problems where the effectiveness of other approaches is limited.

The Code Review Process

Security code review is most successful if it is planned and executed in the context of other security-related efforts such as threat modeling (see Figure 2). Additionally, the results from code reviews can show additional value by improving other security tasks such as testing and design.

Figure 2 Code Reviews
Figure 2 Code Reviews

The value of good threat models for code reviewing can hardly be overestimated. To conduct a successful code review, a reviewer must have a good understanding of a product's goals, its design, and the technologies used for its implementation. The first two areas are also covered by threat modeling, and, although they focus on finding high-level problems, they include research also useful in code reviewing. These two kinds of security efforts are generally complementary; threat modeling helps to identify a critical area of code that then becomes a subject of detailed review. Results from a code review can likewise be used to validate (or question) security assumptions specified in a threat model.

In an ideal situation, a security code review would begin with a review of the quality threat models and design specifications, then move to source code. Before beginning, all development work on code within the scope of the review should be completed. To catch the simpler issues, available security tools should be run against the code before manual review begins. Finally, to avoid finding already-known issues, all previously identified security bugs should be fixed.

The security team should plan code reviewing efforts sufficiently early in the development lifecycle to identify the most suitable time for its execution. An actual plan should also address organizational decisions that might affect the overall return on this investment. For example, there are three likely options for selecting participants in a code review. First, reviewers can be external to the code and product (a code review expert from a penetration testing team). Second, a reviewer can be external to the code but familiar with the product (a developer from another team in the same organization). Finally, reviewers can be familiar with both the code and the product (such as the developer working on the code).

Each option has advantages as well as limitations. In the case of reviewers external to a product, usually skilled security experts are selected and they can make a big difference thanks to their unique experience and perspective. However, in most cases they will have a lower level of understanding of a product's internals and implementation details compared to members of the product team. Developers who wrote the specific fragments of code usually understand them best, but they are also the most susceptible to creator's bias and might therefore overlook significant coding errors.

None of these options is an ultimate solution that can be automatically applied to every case. Code reviewing depends more on human participants than technical solutions and thus it needs to be adopted by the team that is assigned to the effort. Some teams achieve the best results by discussing and analyzing the code in group meetings. In other cases, developers obtain the best results by walking through the code on their own. The team should choose whatever strategy allows participants to be effective in reviewing a product. This general rule applies not only to planning and organizing a code review, but to all challenges that need to be faced in such a process.

Prioritizing Review Efforts

The most important rule for code reviewing is realizing that there is never enough time to review all the code you would like to review. One of the biggest challenges, therefore, is selecting and prioritizing the product's components or code fragments that belong in the primary, or at least initial, scope of analysis. To achieve this goal, it is necessary to understand the design of a product and the role of specific technologies used to implement it. The prioritization is a high-level analysis and it defines the framework for the whole engagement.

The prioritization begins with analysis of threat models and other documentation available for a product. If properly done, a threat model can provide a lot of useful data about relations between different components, entry points, security boundaries, dependencies, and assumptions in the design or implementation. Since the goal of the prioritization is to gain understanding of the implementation details, information from a threat model must then be compared against the code itself, analyzing information about the code's structure, quality, or development process.

Prioritization can be divided into four major tasks, which are presented in Figure 3 along with examples of questions that may be helpful in finding required information.

Figure 3 Code Review Prioritization Tasks and Questions (Close Figure 3)

Understand the Environment and Technologies Used to Implement the Product
Is it a Trusted Computing Base (TCB) component: kernel driver, subsystem, or privileged service?
Is it started by default, co-hosted, or running as a dedicated process?
Is it a reusable plug-in, driver, codec, protocol handler, or file parser?
Does it load and directly invoke other code (third-party libraries)?
What underlying technologies are used (RPC, DCOM, ActiveX, WMI, SQL)?
What languages (C/C++, C#, Visual Basic) and libraries (STL, ATL, WPF) are used?
Enumerate All Sources of Untrusted Input Data (Entry Points)
Is it available via network (TCP, UDP, SMB/NetBIOS, named pipes)?
Does it use IPC communication (LPC, shared sections, global objects)?
Can it be driven programmatically (automation or scripting)?
What resources does it use (registry, files, databases)?
Does it have a UI, parse command-line arguments, or use environment variables?
Does it directly communicate with external hardware (USB driver)?
Determine Who Can Access Entry Points and under What Conditions
Is it a security boundary that allows for potential elevation of privilege?
Is this entry point limited to remote, local, or physical access?
Are there access restrictions on transport, interface, or function level?
Is authentication required (anonymous, authenticated, service, admin)?
What mechanisms are used for authentication, securing secrets, and so on?
Are there any other constraints (allow/deny lists, certificates)?
Include Nontechnical Context of Code
Is it new or legacy code? Is it still in development or in maintenance mode?
Was the security context changed (code moved from user to kernel mode)?
What is the history of security efforts in this product team or organization?
What is the history of the product's security?
What is the security awareness of the team or organization?
What are the results from previous code reviews?
What kind of problems have been detected using automated tools? Where?

The first task is related to understanding the technological context of the code. This context covers not only the specific technologies that are used in a product, but also operating system and third-party dependencies as well as tools used in development. The goal of this task is to identify relationships between a product and other systems, applications, or services. Based on these relationships, it is possible to determine what components a product relies on as well as what other software depends on your product. In the context of security, these relationships determine how a product affects the rest of the system and how it may be affected by it. Some high-risk areas become visible through this process.

The high-risk areas are usually associated with concepts such as security boundaries and potential attack vectors. In practice, it all can be reduced to data that can be controlled by an attacker, which should be considered untrusted, and entry points from which this data can come. The team must analyze each of these entry points in relation to guarantees and assumptions about incoming data. One key question is when and how data is validated. This question can often be related to actual data characteristics (can it change asynchronously or be sent out of order?). It can also be related to characteristics of an entry point, such as whether it is available by default (a network service or programmatic interface) or whether it is created as a result of a user's action (a click or opening a data file).

The characteristics of an entry point are also in the scope of the next big task: analysis of trust relationships and access control. A product may have many different entry points, but its actual attack surface is defined only by those that are accessible to an attacker across a security boundary. Each of the entry points should therefore be investigated to verify who can access it and under what conditions. Unauthenticated access to wide functionality is obviously more interesting for an attacker than entry points accessible only with administrative privileges. In the latter case, the review may be limited to just the piece of code responsible for actual authentication (including the code preceding it).

Last but not least, nontechnical sources can also provide data useful for prioritization of code review. Code is typically created by teams of developers; teams usually change over time as new members join and others leave. Code also has its own security history, sometimes not only in the scope of a specific product. In most cases it can be assumed that newer code is of better quality than legacy code, but there can be exceptions. Talking with developers, reviewing documentation from previous reviews, or sampling code may provide very useful input. To achieve the best utilization of available resources, the security team should use any data that can help reviewers to remain effective throughout the process.

Code Reviewing Tactics

The challenges connected with code reviews of nontrivial applications result primarily from the size and complexity of software. Modern products are created using a variety of different technologies, programming models, coding styles, and code management policies. In practice, the complete and up-to-date documentation is rarely available. A reviewer usually faces much more information than can be processed by a human in an acceptable timeframe. To deal with information overload during the review of complex products, certain practices, referred to as tactics, have been developed. We'll discuss the two most common.

The first tactic, which we call contextual code analysis, is the natural consequence of prioritization and high-level analysis efforts. It relies on precise and context-aware analysis of data flows in areas of code directly involved with processing untrusted data that was passed across a security boundary. The advantage of such a targeted approach is the possibility of focusing on the code that is more likely to be attacked and thus achieve good coverage in the scope of the most critical code paths.

Contextual code analysis can be divided into two passes. The first one starts with an entry point and an understanding of the level of control an attacker has over input data. From this point, the developer conducts a data flow analysis aimed at tracking how variables and data structures change with execution of the program. The analysis is performed on a per-function basis, starting with input arguments, then moving through all execution paths, simultaneously monitoring changes in states of variables and structures.

Every function that is called and takes data controlled by an attacker (coming from an entry point and not yet validated) should be analyzed in the same way. Similarly, any propagation of data to global data structures should be flagged so that every reference to it can later be analyzed. At this point, navigation through the code is aimed at improving understanding of code, discovering its structure, and flagging places for further detailed investigations.

Obviously in the case of security code reviews, one needs to pay special attention to code areas that are more likely to have security problems. Examples of such places are listed in Figure 4.

Figure 4 High-Risk Code Paths for Review (Close Figure 4)

User identification, authentication, and data protection
Authorization and access checks
Code that preprocesses untrusted data (network packet parsers, format readers)
Unsafe operations on buffers, strings, pointers, and dynamic memory
Validation layers ensuring untrusted data is in valid format
Code responsible for conversion of untrusted data to internal data structures
Logic involved with interpreting untrusted data
Places making assumptions about the data itself as well as behavior of its source
Code involved in handling error conditions
Usage of OS resources and network (files, registry, global objects, sockets)
Problematic areas typical to the environment in which the code executes
Usage of problematic API or violations of API contracts

If specific code doesn't process any data that could originate from an entry point or processes data that was already validated, it stops being relevant from a security point of view and the reviewer should move forward to other areas. At the end of the day, a reviewer should have a general understanding of the component's code as well as the list of locations flagged as potentially interesting places in the code. Using this list, the reviewer can begin the second pass of contextual analysis, which is focused on an in-depth investigation of selected areas of code.

We call the second code reviewing tactic pattern-focus code analysis. In this case, a reviewer starts at an arbitrary place in the code and looks for known types of potential security vulnerabilities. Although different supporting tools may be applied, "known" doesn't have to refer to any formal patterns, but rather to the reviewer's individual experience and intuition. For each vulnerability candidate, a reviewer follows up all code paths in order to determine whether the coding error actually represents a vulnerability—processing data that can be controlled by an attacker over a security boundary. If correct validation is identified at any level, the error should not be considered a security vulnerability, although it still may be identified as a defense-in-depth or non-security issue that requires a fix.

Pattern-focused code analysis can be used with limited understanding of an application, since a reviewer can focus on the quality and correctness of selected fragments of code rather than their role or location within a system. This allows for the coverage of a significant amount of code in the scope of specific patterns such as bad code constructs or problematic API calls. However, analyzing potential coding errors in isolation from the context of an entire application usually does not give enough information to determine whether it is a real problem. This may lead to overlooking serious bugs or fixing nonrelevant ones, posing additional long-term consequences to the application. If a bug is described only in local context, it may be tempting to introduce a local fix instead of a more complete validation at a higher level. Such an approach to fixing coding errors may result in redundant validations, increased chaos, decreased performance, and general problems with code management.

Types of Vulnerabilities
Security Resources
  • Trustworthy Computing and SDL
  • MSDN Security Developer Center
  • The Security Development Lifecycle, by Michael Howard and Steve Lipner (Microsoft Press, 2006)
  • Threat Modeling: Uncover Security Design Flaws Using the STRIDE Approach, by S. Hernan, S. Lambert, T. Ostwald, and A. Shostack (MSDN Magazine, November 2006)
  • 19 Deadly Sins of Software Security, by M. Howard, D. LeBlanc and J. Viega (McGraw-Hill Osborne Media, 2005)

During security code reviews, try to maintain an attacker's point of view and look at everything that could be in the attacker's control. Conducting code reviews should not be limited to looking for dangerous APIs and data-copy operations. When it comes to security, the details are significant and history shows that the smallest flaws can be exploitable. This section contains examples of some types of coding errors, but there is no way to cover all of them here. The "Security Resources" sidebar will lead you to more comprehensive sources of information on likely vulnerabilities.

The general concept of security code vulnerabilities is often associated with buffer overflows. In the context of processing untrusted data, coding errors related to range and data type are still a very common source of security problems. This group of vulnerabilities is not, however, limited only to buffer overflows. The buffer overflow condition is connected with copying data to the stack, data segments, or the heap without concern as to their size, which may lead to writing beyond the defined range and hazardous situations resulting from potential overwrites of structures controlling program flow or other sensitive data.

Equally common and severe examples of coding errors related to range validation are out-of-buffer reads and writes. These specific problems often happen when using pointers received from a untrusted source (using pointers as cookies or handle values) or miscalculated indexes and offsets. They allow attackers to access arbitrary places of process memory such as data variables or function pointers and cause changes in program behavior including execution of arbitrary code.

In the scope of calculations, many transformations on numeric type variables (addition, subtraction, multiplication, and so on) may cause the calculation to leave its defined range and silently wrap, leading to integer overflows or underflows. This becomes a problem when the variable is used after transformation and at the same time somewhere else it is used either untransformed or transformed in a different way. As a result, existing checks may turn out to be insufficient, leading to buffer overflows or out-of-buffer reads and writes.

Other problems are related to the sign of numeric type variables. There are many abstract entities that a program deals with that have no defined meaning for negative values. For example, string byte lengths or array element counts do not really make sense as negative values—and using signed types to manipulate these quantities introduces a number of problematic cases.

Type casting (explicit or implicit) is another example of potentially unsafe transformation of data elements. Casting may make items bigger (in terms of bits) and hence may involve the propagation of signed bits into new values. Likewise, type casting could also cause the truncation of a value. Again, problems typically surface when the data item is used in a transformed state in one part of the program while used in an untransformed or differently transformed state in another.

There are specific security code vulnerabilities related to using dynamic memory. Program behavior with uninitialized data elements may be undefined. An attacker may be able to control some uninitialized variables by calling APIs to get or set the uninitialized variables prior to the API that initializes them. Returning uninitialized memory to an attacker can cause a number of problems. Since heap allocations come from a resource that may be shared by different threads servicing different security contexts (clients), sensitive information can leak from one client to the other. This may also provide an attacker with information useful in a broader exploitation of other security flaws, such as aiding the attacker in predicting addresses. Other problems may be related to double-freeing memory, using memory that was already freed, memory leaks, or memory exhaustion. The possible ramifications vary from denial of service and information disclosure to execution of malicious code.

A quite different group of security code vulnerabilities is related to synchronization and timing errors. Good examples are TOCTOU (time-of-check, time-of-use) race conditions. Security and sanitization checks are worthless if performed on data or resources that may be changed by an attacker after the check but before the actual use. Validation has to be performed on private copies of data that cannot change asynchronously. Resources need to be referenced properly to ensure they don't get deleted or replaced.

Multithreaded environments put especially strong requirements on code (services, kernel components) in terms of synchronization of access to shared objects and resources. Programmatic interfaces that allow attackers access to concurrent function calls or their asynchronous cancellation open a window for possible race conditions. Problems caused by missing locks, dropping locks and assuming preserved state, misuse of interlocked operations, or using a disjointed set of locks can lead to unsafe memory operations or deadlocks.

In the case of object lifetime management problems, if code makes a new object available to other threads by manipulating global data (publishing), it needs to be in a consistent (usually completely initialized) state. Proper managing of the lifetime of an object is determined by a proper reference counting mechanism. Broken object destruction and cleanup logic may give an attacker a way to unload code early (such as a plug-in) and free memory that is still in use.

Many security coding errors are not directly related to manipulations on untrusted data, but rather to its actual interpretation or its influence on program behavior and results. The classic examples are found in injection attacks that may occur when data submitted by an attacker is used to parameterize some other content such as a script, Web page, command line, or format string. By using special escape characters or control sequences, an attacker is able to force untrusted data to be interpreted as part of an executed script or a command. Problems arise also when untrusted data is used to construct names and paths to resources that are about to be created or used (files, sockets, registry, shared sections, global objects). Directory traversal or canonicalization issues make code vulnerable to all sorts of redirections that may result either in usage of resources that are under attackers' control or disclosure of sensitive information (for example, user secrets or network credentials).

Security vulnerabilities may also result from faulty assumptions about the data origin and destination (client id, network address, session id, or context handle), the order in which it is coming (network messages, API calls), and the amount (too much or too little data). Often, an attacker is able to control, to some extent, the environment in which code executes by creating named objects before the genuine product creates or uses them (name squatting attacks), filling up the disk space, or either blocking or redirecting network communication. Failure to handle these situations may make code vulnerable to elevation of privilege or denial-of-service attacks. Other common problems result from making assumptions about the technologies used by the code—either about their security guarantees (integrity of the communication channel) or about the inner workings of their APIs (providing incorrect combination of parameters and not checking return values instead of honoring API contracts).

Results from code reviews are often not limited to finding problems classified as code-level problems. Reviewers operate on the level of the actual implementation, and although they can benefit from using design-level documentation, including specifications and threat models, they are not supposed to rely on this data. Thus, code reviews still lead to identification of design-level issues or inconsistencies between the specification and implementation. The types of problems commonly found include exposure of dangerous functionality, incorrect implementation of protocol, usage of custom pseudo-security mechanisms (such as authentication schemes and access checks), or identification of ways to bypass security barriers.

Processing Results

A security code review cannot be considered successful if the security of a product is not improved. Code review should provide output that is useful for development teams to make meaningful changes to a product. Achieving this review-based improvement is connected with the two practical requirements of complete documentation and accurate triaging.

Documentation for each of the identified security code vulnerabilities should contain all details needed to locate and understand the issue. The details should include pointers to flawed code, an explanation of the problem, and justification for why this is a vulnerability. Adding recommendations for a fix is a useful practice, but selecting and preparing the actual solution is the responsibility of the code owners. If any data is missing or it is not clear why a coding error is a security vulnerability, additional research is required from the product team, which is not always possible due to resource or time constraints.

Another important requirement relates to accurate triaging of code vulnerabilities. If the severity of a problem is set too low, it may not be fixed by a product team. On the other hand, if severity is set too high, the problem may be selected for fixing instead of another problem with higher practical impact. The triaging process strongly depends on the quality of the security bug threshold, but also on an understanding of the priorities of code reviewers (who do the triaging) and developers (who act on the results of triaging). As we mentioned earlier, triaging security code errors should not be affected by the availability of exploitation prevention mechanisms.

Wrapping Up

Code reviewing efforts can provide more useful information than just a list of security problems. If possible, reviewers should also document code coverage, confidence in specific code areas, and general recommendations for code redesign and cleanup. Code reviews are also unique opportunities to enrich organizational knowledge, increase security awareness, and improve the effectiveness of security tools. Last, but definitely not least, development teams can use the results of code reviews to help prioritize future product security efforts.

We guarantee that software will always have security vulnerabilities, although the nature of those vulnerabilities and practical impact will change with time. Automated security tools are able to identify more and more coding errors, but some vulnerabilities will still be missed (either not detected or hidden under large numbers of false positives). Manual source code analysis is not a replacement for these tried-and-true tools, but it can often be advantageously integrated with them.

The manual code review approach is expensive, difficult, and highly dependent on the experience and commitment of the participants. However, in many situations the project requires this investment in order to obtain acceptable confidence about the security of a product or its critical components. An experienced human reviewer is still capable of identifying issues that would be missed by tools. As long as a human can be the cause of security problems, a human should also be a part of the solution.

Michal Chmielewski works as a Security Software Engineer in the SWI Attack Team at Microsoft. He specializes in analyzing products for security vulnerabilities.

Neill Clift works as a Security Software Engineer in the SWI team at Microsoft. He is a subject matter expert for kernel and driver issues.

Sergiusz Fonrobert is currently a Security Software Engineer in the SWI Attack Team, responsible for performing penetrating tests of various Microsoft products.

Tomasz Ostwald currently works as a Security Program Manager in the SWI team, where he conducts Final Security Reviews of products released by Microsoft.


Fuzz Testing: Create a Custom Test Interface Provider for Team System

Fuzz Testing

Create a Custom Test Interface Provider for Team System

Dan Griffin
Code download available at: Fuzzing2007_11.exe (301KB)
This article discusses:
  • Visual Studio Team Edition for Testers
  • Fuzz testing
  • Writing a Test Interface Provider
  • Building, testing, and deploying a fuzzer
This article uses the following technologies:
Visual Studio 2005 SDK, VSTS
Contents
What Is Visual Studio Team System?
What Is Fuzz Testing?
Writing a Test Interface Provider
Build
Deploy
Debug
We've Got a Fuzzer ...

W
hile attending the Microsoft® Tech•Ed 2007 conference in Orlando, I had the privilege of working in one of the developer booths at The Learning Center. What struck me the most about that experience was the buzz surrounding the latest Application Lifecycle Management (ALM) tools. There was also plenty of chatter about hot methodologies such as Agile Programming and Test-Driven Development (TDD). And, as a result, there was tremendous interest in the latest ALM suite from Microsoft, the Visual Studio® Team System (VSTS) products.

VSTS offers some great features and extensibility opportunities for testers; that's the focus of this article. Having said that, I should note that this is not an article about TDD! Instead, I'll show how to extend the software testing capabilities of VSTS. I'll use as my example a subset of security testing known as "fuzzing" (more on that later).

What Is Visual Studio Team System?

The server-side component of VSTS is called Team Foundation Server (TFS). The client-side component can be any of the team editions of Visual Studio. For more information about VSTS and TFS, check out msdn2.microsoft.com/teamsystem.

From a technical perspective, the Visual Studio Team Edition (VSTE) products comprise a set of extensions to the Visual Studio 2005 IDE. The different editions provide extensions relevant to a specific discipline—Architect, Developer, Tester, Database Professional. The intent is to wed role-specific functionality to a consistent user interface.

One of the great things about Visual Studio is that it has become more readily extensible with each new release. VSTE is an example of this; however, a better litmus test for product extensibility is how easily it can be accomplished by someone not a part of the company that created it.

To that end, my discussion here of the extensibility of Visual Studio 2005 Team Edition for Software Testers will explore the modification of the existing Test Interface Provider (TIP) sample that's included in the latest Visual Studio SDK. I'll also go into detail about testing, deploying, and debugging the plug-in. My final goal is a TIP that implements a testing strategy that's recently become quite popular, especially among security testers: fuzz testing.

What Is Fuzz Testing?

A good way to explain fuzz testing (fuzzing) is by example. These days, fuzzing is most frequently used to validate two types of data parsers: file and network.

An example of a file parser is Microsoft Word. What would fuzz testing the Word file parser entail? Imagine creating a million Word documents of random length containing random data. Suppose that these documents aren't created by Word itself, but rather by literally piping a random source of binary data into test files on disk. Open each of those random documents in Word and then see what happens.

There are likely to be three potential outcomes of that experiment. First, for the majority of the test files, Word will give some error that the file format is unrecognized or that the file is corrupt and can't be opened. Second, a few of the files out of a million might actually contain some combination of recognizable control and printable characters. Those files will simply open without any problems. Third—and most interesting—it may be that a couple of the files contain data that the document parser doesn't expect. In this case, the program may crash.

To a security tester, that third class of test documents is interesting indeed. If the file parser crashes, that means there's a bug in it. And if there's a bug, it may be exploitable. For example, the bug may result in a stack or heap corruption leading to the crash. In general, this type of bug is of great concern in a networked environment since a wide variety of file types can be distributed as e-mail attachments. Experience shows that most users will open an attachment regardless of its nature.

A problem of similar scope exists in the realm of network parsers. Consider the sheer number of protocols understood by a typical PC, server, or embedded network device: DHCP (Dynamic Host Configuration Protocol), DNS, HTTP, SMB (Server Message Block), SMTP, and so on. Each protocol implementation includes a parser for packet data originating on the network, and each parser is likely to entail some complex logic. This kind of code tends to have lots of edge cases, too, making it difficult to validate.

One approach to testing network protocol code is through fuzzing. However, in this case I'll propose a variation on the just-throw-random-data-at-it approach I described in the context of the Word parser above.

Suppose I want to test a DHCP client. Suppose further that there's a specific portion of the DHCP client code that's particularly nasty, but that piece of code is somewhat protected from the network by another portion of the parser that's considered to be more robust. In this case, fuzzing the DHCP client with purely random data isn't likely to be an efficient use of time, since I expect that the robust piece of code will filter out most of the malformed data. Instead, I take a more targeted approach.

In fact, in my experience, fuzzing is most effective when conducted in this manner. Continuing the DHCP example, instead of bombarding the client with purely random data, I configure my fuzz test (fuzzer) to use a combination of known valid data, intentionally bad data, and random data. The purpose of the known valid data is to bypass the components of the parser that don't interest me. The purpose of intentionally bad data is to take advantage of what I know or suspect to be a specific flaw in the code. And finally, the purpose of the random data is to just see what happens.

During the course of fuzzing, my method evolves as I hone in on suspicious behavior. For example, where some portion of my test DHCP packet originally included random data, I might subsequently use known valid data to get me deeper into the parser. If I discover an error, I might send intentionally bad data to try to exploit it. This heuristic approach gives me the best of both worlds: human intuition plus machine automation.

Fuzzing is indeed a powerful tool, yet it is still in relatively uncharted territory with respect to test coverage. I therefore issue this call to action (to be implemented after reading the rest of the article, of course): pick a technology that you're familiar with, or at least interested in, and write a fuzzer for it. More likely than not, you'll find bugs.

Here's an example: I attended a clever fuzzing demonstration at the RSA conference this year in San Francisco. It inspired me to rush back to my hotel room and write a fuzzer for a technology I had worked closely with. Using a custom fuzzer, I applied the iterative approach described above—start with random data and then fine-tune it—and hit pay dirt within a few hours. As a user, I was able to cause an access violation in code running as LocalSystem.

Writing a Test Interface Provider

The sample code that accompanies this article is an implementation of a simple fuzzer. The intent is twofold: to show fuzz testing in action and to show an example of fuzz testing integration with the VSTE automation framework.

The sample code contains the following four projects:

  1. ConsoleApp—this is the application being tested. The code is short and sweet. This application responds either by outputting a line of text or by throwing an unhandled exception.
  2. MyTest—this is the custom test type provider. The code is mostly unchanged from the sample, except that I renamed MyTestAdapter to FuzzTestAdapter and added all the code to conduct the actual test.
  3. MyTestUI—the C++ code for the custom test user interface. It's unchanged from the original sample.
  4. TestProject—the Startup Project for debugging. It's easy to re-create: just create a new Test Project then add the custom test type.

I discuss these projects in more detail later in this article.

The Visual Studio 2005 SDK is required in order to build the sample. The SDK can be downloaded from microsoft.com/downloads/details.aspx?familyid=51A5C65B-C020-4E08-8AC0-3EB9C06996F4. The current version of the SDK is 4.0, dated 2/28/2007. It includes the original sample VSTE TIP upon which mine is based; however, I had some problems getting the original sample to build, install, and run. While the changes required to turn the existing "MyTest" sample into the "FuzzTest" sample discussed in this article were minor, the effort required to get to the point of debugging the running plug-in was not. Help from the product team, and a lot of trial and error, solved those problems and I'll describe the steps in detail in a moment.

Again, the changes that I made to the original MyTest sample to turn it into the FuzzTest sample that accompanies this article are easy to grasp. In fact, running windiff.exe with the –T switch to compare the original and modified sample directory trees shows a short list of changes that is both interesting and instructive.

The first necessary change was to get two files that are missing from the base MyTest sample, at least in the version of the Visual Studio SDK referenced previously. The files are mytest.ico and mytestui.rc. Both files were provided to me by the VSTS product team and are included in my FuzzTest sample code. They both go into the solution mytestui subdirectory.

The second code change was to the mytestui project file. Specifically, a post-build command was using a relative path and was doomed to fail when I attempted to build the sample project in a directory other than its default location (that is, in the SDK installation tree). The command in question launches the Command Table Configuration compiler (ctc.exe), a tool included with the Visual Studio SDK. For more information about CTC files, see msdn2.microsoft.com/bb165048.

To address this problem, I modified the project to launch ctc.exe from its default absolute path—an approach that admittedly has its own shortcomings (for example, if the SDK isn't installed to the C: drive)—but at least you'll know what to look for!

As an aside, I mention the above changes since they're relevant not only to the FuzzTest sample, but also for getting the original sample built in case additional experimentation is desired. There are some important deployment and debugging issues to be aware of as well, but I'll get to those later in the article.

In turning the base MyTest sample into a new test plug-in, the next step was to replace the original MyTestAdapter class (in mytest\myhostadapter.cs, which I removed) with the new FuzzTestAdapter implementation in mytest\fuzztestadapter.cs. Copying those two files to a temporary directory and running windiff.exe on mytestadapter.cs and fuzztestadapter.cs shows essentially all of the core code changes required to create the new TIP. And in fact, the majority of the interesting changes exist within the Run function (see Figure 1).

Figure 1 Implementation of the IBaseAdapter::Run Method (Close Figure 1)
[SuppressMessage("Microsoft.Design", "CA1031")]
public void Run(ITestElement testElement, ITestContext testContext)
{
MyTestAssertHelper.ParameterNotNull(testElement, "testElement");
MyTest test = testElement as MyTest;
MyTestAssertHelper.ParameterNotNull(test, "testElement");
MyTestResult result = new MyTestResult(
       new ComputerInfo(Environment.MachineName), m_runId, test);
Stopwatch timer = null;

try
{
// Start the timer for this test.
timer = new Stopwatch();
timer.Start();

// This is a FUZZ Test: Loop until the test breaks, then report back
// loop through unicode chars and pass into method
bool ErrorFound = false;
char ch;
for (int i = 0; i < char.MaxValue; i++)
{
       ch = Convert.ToChar(i);
       string testParam = ch.ToString();
       string CommandLine = test.CommandLine.Replace(
              "<FuzzTestString>", "\"" + testParam + "\"");
       int spaceIndex = CommandLine.ToUpper().IndexOf(".EXE ") + 4;

       Process p = new Process();
       p.StartInfo.FileName = CommandLine.Substring(0,spaceIndex);
       p.StartInfo.Arguments = CommandLine.Substring(spaceIndex+1);
       p.StartInfo.UseShellExecute = false;
       p.StartInfo.CreateNoWindow = true;
       p.StartInfo.RedirectStandardInput = true;
       p.StartInfo.RedirectStandardOutput = true;
       p.StartInfo.RedirectStandardError = true;
       p.Start();

       StreamWriter sIn = p.StandardInput;
       StreamReader sOut = p.StandardOutput;
       StreamReader sErr = p.StandardError;
       sIn.AutoFlush = true;
       sIn.Write(Environment.NewLine);
       string OutputString = sOut.ReadToEnd();
       string ErrorString = sErr.ReadToEnd();
       p.WaitForExit();

       if (ErrorString != "")
       {
       // Record the process exit code.
       ErrorFound = true;
       result.ProcessExitCode2 = -1;//failed
       result.Outcome = TestOutcome.Failed;
       result.ErrorMessage =
              "Test failed with input: " +
              CommandLine +
              ", Error Message: " +
              ErrorString;
       timer.Stop();
       break;
       }
}

if (!ErrorFound)
{
       // all tests succeeded:
       result.ProcessExitCode2 = 0; // success
       result.Outcome = TestOutcome.Passed;
       timer.Stop();
}
}
catch (Exception ex)
{
       result.Outcome = TestOutcome.Failed;
       result.ErrorMessage = ex.ToString();
}
finally
{
       if (timer != null)
       {
       timer.Stop();
       result.Duration = timer.Elapsed;
       }
}
testContext.ResultSink.AddResult(result);
}

Analyzing the code for the function Run in Figure 1 reveals how the new FuzzTestAdapter works. In short, when a test of type FuzzTestAdapter is to be executed in VSTE, the first step (after some object initialization at the very beginning) is to start a timer. Test run duration isn't necessarily the most interesting statistic relating to a fuzz test (as opposed to, say, a performance test), but it's supported as a built-in feature of the TestResultMessage class, so I left it in.

The next step in Run is to extract the command-line configuration option for the target test instance, which indicates how to run it. Notice that, following the full path to the target test binary, the plug-in is expecting to find a placeholder string, "<FuzzTestString>". The string as a whole is retrieved via the CommandLine member of the MyTest class. The purpose of that string is to expose to the user a convenient way to reuse the TIP for different tests. This becomes even more convenient in the context of my Fuzzer TIP, since the command line changes each time the test loops.

Continuing with the description of the Run method, the next step is to instantiate a System.Diagnostics.Process object to launch the test program on the local machine. If, as a result of the test input (or for any other reason), the test process throws an exception or returns an error, a test failure will be flagged. Note that I adorned my implementation of the Run method with a [SuppressMessage] attribute. That attribute is necessary in this context since the execution of the test process is wrapped by a handler that will catch all exceptions. This is generally considered bad practice, and is by default flagged as such by the VSTS static code analyzer, but it's required in this case to ensure that all non-positive test completion scenarios are caught and tracked as failures.

To wrap up the discussion of Run, and to describe the specific type of fuzz test implemented by my TIP, observe in Figure 1 that most of the logic described in the previous two paragraphs is wrapped in a loop that iterates through each possible value that can be taken by the managed char type. The net result is that the test binary (ConsoleApp.exe in this sample) is called with each possible char value, continuing until either the loop completes or the test program fails to correctly handle one of the values in that range. Naturally, for the sake of the demo, I've ensured that the test will fail; otherwise, it wouldn't be an interesting example!

The ConsoleApp test demonstrates how fuzzing can be used to expose errors even in simple programmatic interfaces. In this case, the interface was written based on an implicit and risky assumption about the range of character values to be handled. The following snippet, adapted from the ConsoleApp code, shows this assumption:

Int32 arraySize = Math.Max(Math.Max('Z', 'A'), Math.Max('z', 'a')) + 1;
charCountArray = new Int32[arraySize];
char c = inputString[0];
charCountArray[c] += 1;

In this code, the inputString variable is the second command-line argument received from the FuzzTestAdapter. Any character value outside the ASCII upper- and lower-case alphabet range will result in an "array index out of bound" error. As the fuzzer loops through its character values, it will quickly arrive at one that causes the code to fail. When that happens, the ConsoleApp process terminates with an unhandled exception. That result is retrieved by the fuzzer via its Process p object and returned to VSTE.

Once the MyTestAdapter class in the original sample code was replaced by the FuzzTestAdapter class, all that remained to do in my implementation was to rename references to the former by references to the latter.

Now that I've explained the changes required to turn the original TIP sample into my FuzzTestAdapter, let's look at building, deploying, and debugging.

Build

In order to build the sample FuzzTestSample.sln solution, open it in VSTE and then select Build | Build Solution from the top-level menu. If you want to just start with the basic sample solution included with the SDK, you'll first need to grab the two missing files that are included with the code download accompanying this article.

Deploy

Both the original TIP sample and the updated fuzzer version include a deployment batch script. The script is the recommended way to install the TIP binaries for testing. The script assumes that it's being run on the same machine as the binaries were built and from within a Visual Studio command window.

Before running the script (but after building the solution, of course), I've found that it's safest to shut down Visual Studio for the deployment phase. This is admittedly inconvenient, but if I didn't perform this step, some of the file copy operations during deployment would fail.

To run a Visual Studio command window, select Start | All Programs | Microsoft Visual Studio 2005 | Visual Studio Tools | Visual Studio 2005 Command Prompt. Then run the deploy.bat setup script. I made minor changes to the script; you may find that the original one needs some tweaks, too, depending on where you run it. Take a look at the relative paths used in the copy and xcopy commands for deploying the binaries. Also, note that the last command performed by the script is a call to regpkg.exe, based on a relative path to the Visual Studio Integration Tools directory in the SDK. I changed this to an absolute path using the %ProgramFiles% Windows environment variable, based on the assumption that the default installation directory for the SDK was used.

Finally, don't forget this last step, since it's not part of the batch file! From within the Visual Studio command window, run the devenv /setup command. This command ensures that your plug-in's resource metadata is merged with the metadata of other installed Visual Studio packages and completes the installation of the new plug-in.

In preparing this article, I asked the Visual Studio product group how the TIP is actually consumed. Each Test Type must define a TIP, which will be instantiated by TMI (Test Management Interface). TMI will use this TIP to load tests from storage, create new tests, save tests, and interpret results.

Debug

Once deployment is completed, the next step is to create and run a test of the new type. Here are the steps I used to accomplish this:

  1. Reopen the FuzzTestSample solution in VSTE.
  2. Ensure that TestProject is set as the default start-up project. To do this, right-click on TestProject in the Solution Explorer window and select Set as Startup Project. After doing this, the TestProject name should appear in bold lettering in Solution Explorer, as shown on the right-hand side of the screenshot in Figure 2.
  3. Right-click on TestProject again in Solution Explorer and select Add | New Test.
  4. In the New Test window that appears, you should see a test type entitled My Test. That's the new test type from the sample TIP, and its presence implies that the registration steps were successful. Select the My Test icon and click OK.
  5. In the Command Line field of the new test, replace the default entry with the full path to the ConsoleApp.exe test program, appended with the expected command-line values (as discussed in the previous section regarding the Run routine). For example, you might enter "c:\projects\FuzzTestSample\ConsoleApp\bin\Debug\ConsoleApp.exe ProcessString <FuzzTestString>". See the left-center pane in Figure 2.
  6. Run the test by pressing the F5 key.

Figure 2 VSTE with Sample Test Result
Figure 2 VSTE with Sample Test Result(Click the image for a smaller view)
Figure 2 VSTE with Sample Test Result
Figure 2 VSTE with Sample Test Result (Click the image for a larger view)

This approach has the benefit of running the TIP in the debugger. For example, prior to pressing F5, I can set a breakpoint on the Run function in MyTest\MyTest\FuzzTestAdapter.cs and watch it launch ConsoleApp.exe with the various character values. Of course, in production, I wouldn't expect the plug-in to be run in the debugger since that slows down test execution considerably.

This particular test takes a few moments to complete. Once it's complete, the test failure pane—shown at the bottom of the screenshot in Figure 2—becomes visible. From the error text shown in the test failure, it's clear that a non-alphabetic character caused an IndexOutOfRange exception, just as expected.

It's worth mentioning the method I learned for making code changes to the custom test type after a successful initial deployment. Based on experimentation, this sequence of steps appears to be the minimum required for redeploying the plug-in in such a way that it can be debugged again.

  1. Make the desired code changes and rebuild the solution (incremental is fine).
  2. Shut down Visual Studio.
  3. Copy the updated file, Microsoft.VisualStudio.QualityTools.Samples.MyTestTIP.dll, from the solution output directory to its deployed location. By default, the latter is C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies.
  4. Reopen the solution in Visual Studio.
  5. Add a new test (for example, MyTest2.mytest).

We've Got a Fuzzer ...

Effective automation is considered the ultimate goal of most software Quality Assurance teams. And discovery of many classes of defects, including security bugs, has proven particularly difficult to automate. A heuristic approach, such as the adaptive fuzzing method described at the beginning of the article, combined with an automation framework such as Visual Studio Team Edition, provides the best of both worlds. That is, human intuition, plus a machine's tolerance for boring repetition, equals good test coverage!

I'd like to thank Euan Garden and Jeff Wang at Microsoft, as well as Juan Perez at Personify Design, for their input into this article.

Dan Griffin is a software security consultant in Seattle, WA. He previously spent seven years at Microsoft on the Windows Security development team. Dan can be contacted at jwsecure.com.


Editor's Note: Why Go to Extremes?

Editor's Note
Why Go to Extremes?
Howard Dierking

"Is your system secure?" This question, along with its cousin, "How secure is your system?," seems to be one of the funnier questions we are typically asked by business and IT stakeholders alike. The question is funny, not because security isn't important—to the contrary, in this day and age, security has never been more important—but because it is too general to have any measurable meaning.

In both variants, the most concrete answer you could ever really give is: "more secure than some, less secure than others." Of course, this answer is not very satisfying to a business, and as such, it is never given. Instead, we often end up defining security in terms of rather extreme absolutes. In one extreme, we focus solely on the business requirements of the system, relying on the IT department to supply a checklist of security-related tasks that we implement before deploying the system. In the other extreme, we spend countless amounts of time and energy researching and securing against every possible exploit that has ever been attempted in the history of the software industry—far surpassing the most stringent of DoD standards—and many times subjugating the needs of the business to the needs of our paranoia.

For this year's security issue of MSDN Magazine, I'm very happy to see an emphasis placed on a pragmatic approach to creating an effective, measureable security strategy. Michael Howard leads off with a discussion of some practical best practices he has learned over the past five years in building secure software at Microsoft. Following Howard's discussion of best practices, Michal Chmielewski, Neill Clift, Sergiusz Fonrobert, and Tomasz Ostwald discuss techniques for integrating security-related activities—both automated and manual—more deeply into the existing development lifecycle. Dan Griffin extends this discussion with a more concrete example of automated integration by developing a fuzz testing add-in for Visual Studio Team System. Finally, for those of you who just don't feel complete as developers without looking at assembly language, Adel Abouchaev, Damian Hasse, Scott Lambert, and Greg Wroblewski dive deep—really deep—into the different conditions that can cause your applications to crash, and how each of those conditions may additionally create security vulnerabilities.

One of the themes that seems to reoccur throughout many of this month's articles is that security concerns are constantly evolving and are based on a realistic assessment of threat—which is why the emphasis in this issue is more about creating a solid security strategy than it is on how to "do security" for an application. When you look at security in this light, security concerns look more like business requirements than static, infrastructure concerns—and perhaps if we can look at them this way, we can figure out how to create a reasoned, measurable security strategy, and thereby avoid going to one extreme or the other.

Speaking of measurable, one other quick note: we are constantly trying to improve the quality of both the content and the planning of MSDN Magazine. And, among other things, we rely on the ratings system found on the online version of each article in the issue. You can have a very direct impact on the magazine by simply going to the online version of an article you read here and rating it using the rating control. Your opinion matters.

Code safe! —Howard

Thanks to the following Microsoft technical experts for their help with this issue: Eric Bidstrup, Joe Binder, Todd Brooks, Claudio Caldato, John Durant, Steve Fox, Matt Gibbs, Ed Hintz, Michael Howard, Richard Johnson, Ivan Medvedev, Dave Reed, Bruce Taimana, Chris Tavares, Mads Torgersen, and James Whittaker.


Toolbox: Updated grid controls, open-source comparison tools, and more

Toolbox
Updated grid controls, open-source comparison tools, and more
Scott Mitchell

Contents
Upgrade Your Windows Forms Grid
Add Professional-Looking Date Textboxes to Your Web Site
Quickly Compare and Merge Files
A Guide to Microsoft ASP.NET AJAX

Upgrade Your Windows Forms Grid

The grid is probably the most frequently used advanced user interface element as it is the easiest and most straightforward way to display and edit tabular data. Since the early days of Visual C++® and Visual Basic®, Microsoft has provided developers with a free, simple grid control. But developers can also take advantage of the grids that are available from third-party vendors.

One third-party vendor for Microsoft® .NET Framework Windows® Forms grid controls is FarPoint Technologies, Inc., which offers a grid component named Spread for Windows Forms 3. To add Spread for Windows Forms to your .NET application, simply drag an instance from the Visual Studio® Toolbox and drop it onto your form. Its initial data can be specified manually, programmatically, or via databinding.

Adding a Spread for Windows Forms grid to your application is like adding an instance of Microsoft Excel®. Like Excel, Spread for Windows Forms can include multiple sheets and formulas, have merged cells and cells that span columns, provide column-level sorting, and more. With one line of code you can import an Excel spreadsheet into a grid. Similarly, grid data can be exported to an Excel spreadsheet or XML file.

Customizing a Spread for Windows Forms grid in Visual Studio is particularly easy thanks to the Spread Designer. This standalone program, which can be launched from the Properties window in Visual Studio, enables developers to specify the appearance, layout, and format of their grids.

Grids are as routine a fixture in user interface design as buttons and labels. And some third-party grid products, such as Spread for Windows Forms, offer developers functionality beyond that provided by Microsoft.

Price: $899 per developer license.

www.fpoint.com

Spread for Windows Forms Grids Are Customizable
Spread for Windows Forms Grids Are Customizable (Click the image for a smaller view)
Spread for Windows Forms Grids Are Customizable
Spread for Windows Forms Grids Are Customizable (Click the image for a larger view)

Add Professional-Looking Date Textboxes to Your Web Site

Large, commercial Web sites that prompt users for date values—such as travel reservation sites—often provide a textbox with a calendar icon next to it. Clicking this calendar icon displays the current month in a popup calendar and enables the user to select a date by simply clicking on it.

Such rich date entry user interfaces are found less often on intranet sites and on smaller Web sites, in part because ASP.NET does not include any built-in date-related Web controls. The good news is that there are many third-party date controls available. One such product is Peter's Date Package (version 2.0.4), created by Peter Blum, which comprises 24 ASP.NET Web controls. The flagship control is the DateTextBox, which displays a textbox with a calendar icon that, when clicked, shows a popup calendar interface. The calendar displays the current month with icons to display the dates from the previous or next months. The pop-up also includes a month and year picker, which expedites entering dates far in the future or past. Furthermore, the control automatically emits client-side script that prevents the user from typing non-date values into the textbox (such as letters or non-date related punctuation).

In addition to the DateTextBox control, Peter's Date Package includes a standalone month and year picker, an AnniversaryTextBox that lets users specify only a month and day, and a TimePicker control. There is also an array of date-specific validation controls. The UnselectableDatesValidator allows a developer to indicate a set of invalid dates, while the DateCompareValidator ensures that the dates entered into two date controls maintain a certain relationship, such as the first control's value being less than the second's.

Best of all, Peter's Date Package is a cinch to use. Configuring a new or existing Web site to use the date controls takes less than a minute, and including these rich date controls in a Web page is as simple as drag and drop. If you get stuck, Peter Blum is quick to answer questions posted to the support forums.

Price: $90 per Web server.

www.peterblum.com/DateControls

DateTextBox is the Flagship Control of Peter's Date Package
DateTextBox is the Flagship Control of Peter's Date Package (Click the image for a smaller view)
DateTextBox is the Flagship Control of Peter's Date Package
DateTextBox is the Flagship Control of Peter's Date Package (Click the image for a larger view)

Quickly Compare and Merge Files

Since developers work with the source code for applications as a collection of files, we often need to compare different versions of files or folders and, perhaps, synchronize them. For example, you may need to determine what files have been modified since the last build, or update the production Web site's pages with the modifications made on the staging server. Programs that find differences in files have been around for decades—Windows has long included such a program, called WinDiff. Furthermore, virtually all source code control providers bundle some sort of WinDiff-like program to facilitate comparing different file versions under source control. But some developers want more capabilities than WinDiff and source code control programs offer.

WinMerge (version 2.6.8) is an open-source file and folder comparison and synchronization tool that is free, fast, and feature rich. When launched, WinMerge prompts you to select two files or folders to compare. You can also specify a file filter and line filter at this time. When including a set of files in a folder (or subfolders), the file filter indicates what files to include or exclude. For example, Web developers might want to use a filter that includes HTML and ASP.NET Web pages, but excludes GIF, JPG, CSS, and JavaScript files. You can create your own file filters or use one of the several that ships with WinMerge. Line filters can be used to exclude comparing lines of text that match a particular regular expression.

If you are comparing files from two different folders, WinMerge lists the files in each folder and whether or not they are identical. Clicking on one of the listed files displays a window showing the contents of the two files, using colors to indicate their differences.

WinMerge also makes it easy to synchronize files. You can step through the differences one at a time and choose whether to make the left file's difference the same as the right's, or vice versa. You can also make one file exactly like the other with the click of a button.

In addition to its intuitive graphical user interface, WinMerge can be executed from the command line and can be integrated with the Windows shell.

Price: Free.

www.winmerge.org

WinMerge Has an Intuitive Graphical Interface
WinMerge Has an Intuitive Graphical Interface (Click the image for a smaller view)
WinMerge Has an Intuitive Graphical Interface
WinMerge Has an Intuitive Graphical Interface (Click the image for a larger view)

A Guide to Microsoft ASP.NET AJAX

AJAX is used by many to build interactive Web sites that offer a streamlined, desktop application-like user experience. AJAX-enabled sites use JavaScript to asynchronously communicate with the Web server and to seamlessly update the browser's interface accordingly. There are a variety of third-party and open-source AJAX frameworks, and now Microsoft has released its own AJAX framework, ASP.NET AJAX (ajax.asp.net).

Most AJAX-related books offer a rather exhaustive low-level look at the technologies involved. But that level of detail is unnecessary for a developer who wants to quickly get started building AJAX-enabled ASP.NET apps. These developers will find Matt Gibbs and Dan Wahlin's Professional ASP.NET 2.0 AJAX (Wrox Press, 2007) to be just what the doctor ordered. The book focuses on core AJAX concepts such as asynchronous postbacks, the browser's Document Object Model, and making HTTP requests via script—and each is explained in terms familiar to ASP.NET developers. Working with the Microsoft ASP.NET AJAX library is also explored, as is using the UpdatePanel control, the ASP.NET client library, ScriptManager, and the controls in the AJAX Toolkit.

Professional ASP.NET 2.0 AJAX also showcases tips and techniques for debugging. The techniques involved in building AJAX-enabled Web applications complicate the testing process. Since many AJAX-related functions occur in a Web browser, developers lose the rich debugging support available with server-side code. The chapter on testing examines ASP.NET AJAX's debugging-related classes and helpful tools such as Script Debugger from Microsoft, Eric Lawrence's Fiddler, and Nikhil Kothari's Web Development Helper. Additionally, the book explores the challenges involved in deploying any AJAX-enabled ASP.NET applications.

Price: $39.99.

www.wrox.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. You can reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: Managing Object Lifetime

CLR Inside Out
Managing Object Lifetime
Tim Fischer

Contents
Garbage Collection and Finalization
Lifetime Management in COM
.NET Deterministic Finalization
Finalization in a COM Library
Memory Leaks

A
lthough the Microsoft® .NET Framework provides a managed execution environment, it is important to use resources carefully, consider lifetime management, and write code that addresses these concerns correctly. In this column, I have set out to help developers who are consuming .NET or COM classes to better understand lifetime management, recognize situations where you can run into resource traps, and know how to solve these issues.

Note that, for clarity, the sample code shown in the column only works for single-threaded applications. Reusable thread-safe code for .NET resource management is available in the LifeTimeScope.net project on CodePlex (see codeplex.com/lifetimescope).

In .NET programming, an object is created with the new keyword and allocated on the managed heap. The object is alive until the garbage collector finds out that the object is no longer reachable from a strong root reference via a path of strong references. Each program has a main method and an associated class as well as static classes that all might contain local variables, static members, or even events. These static or local references are considered to be root references within your program (see Figure 1). Ordinary .NET Framework references are known to be strong references. Object lifetime is determined by the path of strong references that point from a root reference to an object.

Figure 1 Strong References to Objects on the Heap
Figure 1 Strong References to Objects on the Heap

In some cases, such as when building a fast cache or a hash into an object graph, it might be interesting to have additional references to objects that do not change the lifetime of an object. For this purpose the .NET Framework offers an additional class called WeakReference that allows you to implement a weak reference between objects. This weak reference is ignored when determining the lifetime of an object.

Events can also be strong root references and as such can contribute to the strong reference path and thus affect the lifetime of an object. Ordinary events in the common language runtime (CLR) 2.0 are bidirectional strong references between the event source and the listener and as such can keep an object (either source or listener) alive that otherwise should be dead already. This is why a WeakEvent class was added in the .NET Framework 3.0. It is important that you become familiar with the WeakEvent pattern, which is not yet well known, but is required to implement the Observer pattern successfully. The WeakEvent pattern has been used in the Windows® Presentation Foundation (WPF) Data Binding implementation to prevent leakage due to data binding.

Garbage Collection and Finalization

Let's look at how the CLR determines whether an object is still alive and what happens when it isn't. In the November 2000 issue of MSDN® Magazine (msdn.microsoft.com/msdnmag/issues/1100/GCI), Jeffrey Richter describes the procedure. Each .NET process runs a separate thread that is used by the garbage collector. When the garbage collector runs, all other threads are paused. The garbage collector analyzes the memory structures and references to find out which objects are dead. These objects are then queued for a process called finalization. In a second phase, the finalization process is executed on the separate thread used by the garbage collector and the Finalizer method is called on each object.

There are two important points to note in this process. First, the Finalizer is called from a different thread than the thread that created the object. Second, the Finalizer is called at some point in time but not necessarily close to the time the object actually went dead (this can be noticed when the system is stressed, but memory is available).

The garbage collector is exposed in the static System.GC class. You can force garbage collection to run and let it identify the dead objects and queue them for finalization by calling GC.Collect. You can also force finalization to happen for all queued objects by calling GC.WaitForPendingFinalizers. However, there are two issues with calling WaitForPendingFinalizers: this is a time-consuming process and the Finalizers are called from another thread. This can lead to performance and threading issues if, for example, you need to release a reference to a system resource and plan to do this in your Finalizer—some unmanaged resources are thread-dependent. (In that case, please read on and learn about the Dispose pattern.)

As the .NET Framework team already identified the need for a timely destruction of objects and also the need for allowing the destruction code to be executed on a specific thread, they came up with a solution known as the Dispose pattern. For details, see the CLR Inside Out column in the July 2007 issue of MSDN Magazine at msdn.microsoft.com/msdnmag/issues/07/07/CLRInsideOut.

The Dispose pattern is a convention that should be implemented by every object that has resources and needs timely destruction. It is accomplished by implementing the IDisposable interface and releasing resources in the implementation of a method called Dispose exposed by the interface. However, just like in the unmanaged world, it is still the burden of the programmer to make sure the Dispose method is called at the right time in the client code. Brian Harry has a posting on his public forum that explains why the team came up with this solution and why other interesting solutions could not be applied successfully (see discuss.develop.com/archives/wa.exe?A2=ind0010A&L=DOTNET&P=R28572). The Dispose pattern must be correctly implemented for all objects that own resources.

Lifetime Management in COM
Lifetime Management Resources
  • MSDN .NET Framework Developer Center
  • The WeakEvent Pattern
  • LifeTimeScope.net Helper Classes

In the COM world, lifetime management works via reference counting. Each COM object has an internal count of client references that point to it. These counters are incremented and decremented by the client developer by calling IUnknown::AddRef and IUnknown::Release. Note that AddRef increases the counter and should be called when a reference to the component is set while Release decreases the counter and should be called when the reference has been destroyed. When the counter goes back to 0, the object can be destroyed.

The calls to AddRef and Release were eased by the use of SmartPointers in Active Template Library (ATL) or by generating the necessary code by the compiler/runtime such as in Visual Basic® 6.0. Events were only strong pointers in one direction and weak pointers in the other direction.

The two major issues with reference counting are performance loss and the potential to be unable to detect dead objects that are part of a cyclic-reference scenario. The performance loss is generally acceptable in COM because it's typically used as a technology to combine larger components. The .NET Framework, however, is used on the class level and thus the performance impact would be too great if reference counting were added to the CLR.

If you use a COM component from managed code, you use tlbimp.exe to generate a so-called runtime callable wrapper (RCW), which exposes the functionality of the COM component to the managed world. Note that tlbimp.exe is executed in the background when you reference a COM component in Visual Studio®. You can easily identify RCWs as they are displayed as System.__ComObject in the debugger. They are .NET types that hold one unmanaged pointer to the COM object and call AddRef and Release on the object. AddRef is called once the component is created and Release is called when the RCW is finalized.

But when is the RCW finalized? You do not know, so you need a method to call Dispose on the RCW. Unfortunately, the generated RCWs do not implement IDisposable. Currently, the equivalent to calling the missing Dispose method is to call:

System.Runtime.InteropServices.Marshal.ReleaseComObject(MyCOMObject)

.NET Deterministic Finalization

In many cases you need to make sure that the resources you use are released as soon as possible. The ideal way is through the using keyword in C# to ensure that the Dispose method is called on the object once program flow leaves the scope indicated by the curly brackets:

using (FileStream theFileStream = File.Create("C:\\hello.txt"))
{
    string s = theFileStream.Name;
}

In addition, it's worth noting that C++/CLI automatically calls the destructor of local variables when exiting a method. In these scenarios, you can generally assume that the Dispose method is called automatically at the end of the block or method.

However, these approaches do not work when the usage scope of the resource is not local. Assume you call another method in the preceding code that uses the FileStream object as an input parameter. That method may not call Dispose or use the using keyword. In the context of a multithreaded server sharing resources between threads, this might become even more complex.

Unfortunately C# does not provide a simple construct equivalent to the using keyword that can be used across scopes or threads. This makes reusing code and routines difficult as the responsibility of calling Dispose must be negotiated between components to ensure it is done only once, and at the right time.

Beware: if you do not call Dispose explicitly, then the Finalizer will do it for sure if the object implements the Dispose pattern properly. However, you risk running out of resources under high loads because you do not know when the Finalizer will be called. You also risk having the Finalizer fail when it is called from another thread than the one that created the object. This can become an issue if you try to release unmanaged resources or COM resources in a Finalizer.

One possible solution is to build a generic wrapper class around any type implementing IDisposable. The wrapper class returns an IDisposable interface, but internally sets a reference counter before delegating the call to the component. The code shown in Figure 2 uses a new extension method, called AddRef, to get a corresponding wrapper object that contains an internal reference counter for a specific object.

Figure 2 An IDisposable Wrapper (Close Figure 2)
static void Main(string[] args)
{
    FileStream theFileStream = File.Create("C:\\hello.txt");
    // Unscoped use of RefCounted
    using (theFileStream.AddRef())
    {
        a(theFileStream);
    }

    System.Console.Read();
}

static void a(FileStream inFileStream)
{
    // Unscoped use of RefCounted
    using (inFileStream.AddRef())
    {
        // do something
        string s = inFileStream.Name;
    }
}

When AddRef is called, the code looks into a hash table to see if the object already has a wrapper. If so, it returns the wrapper and increases the counter; otherwise it creates a new wrapper and adds it to the hash table. When the program reaches the end of the using block, Dispose is called automatically and it decrements the counter. In case the counter goes to zero, the Dispose call is propagated to the underlying object.

The implementation of the extension method and wrapper is shown in Figure 3. For clarity, this code is not designed to be thread safe.

Figure 3 AddRef Extension and Wrapper (Close Figure 3)
public static class ReferenceCountingExtensions
{
    public static System.Collections.Generic.Dictionary<object,object>
        theHashOfReferenceCountedClasses =
        new System.Collections.Generic.Dictionary<object,object>();
    public static RefCounted<T> AddRef<T>(this T resource)
        where T : IDisposable
    {
        object theObject=null;
        theHashOfReferenceCountedClasses.TryGetValue(resource,
            out theObject);
        if (theObject == null)
        {
            theObject = new RefCounted<T>(resource);
            theHashOfReferenceCountedClasses[resource] = theObject;
        }
        else
        {
            ((RefCounted<T>)theObject).AddRef();
        }
        return (RefCounted<T>) theObject;
    }
}

// Base Class for a RefCountedUnmanagedResourceProtectingClass
public class RefCounted<T> : IDisposable
{
    private T theResource;
    bool IsFinalDisposed = false;
    int RefCounter = 0;
    public RefCounted(T inResource)
    {
        theResource = inResource;
        System.Threading.Interlocked.Increment(ref RefCounter);
    }
    public RefCounted<T> AddRef()
    {
        System.Threading.Interlocked.Increment(ref RefCounter);
        return this;
    }
    public T Resource
    {
        get
        {
            return theResource;
        }
    }
    ~RefCounted()
    {
        if (!IsFinalDisposed)
        {
            Dispose();
        }
    }
     public void Dispose()
    {
        System.Threading.Interlocked.Decrement(ref RefCounter);
        if ((RefCounter == 0) && (!IsFinalDisposed)) FinalDispose();
    }
    virtual public void FinalDispose()
    {
        ((IDisposable)theResource).Dispose();
        IsFinalDisposed = true;
    }
}

Make sure that your Finalizers really run to the end. The .NET Framework runtime provides a concept called Constrained Execution Regions and the CriticalFinalizerObject that can help you achieve this.

Finalization in a COM Library

Let's look at a scenario where you use a COM object model from C#. You can use any COM component easily by creating a reference to its registered type library. Let's assume you have a COM component exposing a telephone API (TAPI) and use it in the following lines of code—see if you can find a bug related to lifetime management here:

void foo (CTAPIApplication TAPIApplication){
    string s = TAPIApplication.Ports[0].SpeechListener.GetName();
}

The bug is that a Ports collection was acquired from the object model, but its ReleaseComObject method was not called. The same is true for SpeechListener. You may argue that it is not safe to assume that foo should release the ComObject, and you are right; it is not clear from this code who should release the object and when. So what can you do?

The solution is again to use the AddRef Method, but you need to modify the RefCounted class to understand that it must call ReleaseComObject in case it is a ComObject. Figure 4 shows the enhanced FinalDispose method. In addition, where T : IDisposable was removed from the call to AddRef<T> because there is no common interface that IDisposable and COM Objects share:

Figure 4 FinalDispose Method (Close Figure 4)
virtual public void FinalDispose()
{
    if(theResource.GetType().IsCOMObject)
    {
        Marshal.ReleaseComObject(theResource);
    }

    else
    {
        ((IDisposable)theResource).Dispose();
    }

    IsFinalDisposed = true;
}

public static RefCounted<T> AddRef<T>(this T resource)

Now you can rewrite the call into the telephone API as shown in Figure 5. While this code still is not very handy, it is very safe because the using statement ensures that ReleaseComObject really gets called even if an exception is thrown at some point. Adding another helper class called LifeTimeScope and changing the return type of AddRef makes the code even more readable:

Figure 5 Calling into COM (Close Figure 5)
void foo(CTAPIApplication aTAPIApplication)
{
    using(PortsCollection aPortsCollection =
        aTAPIApplication.Ports.AddRef())
    {
        using (Port aPort = aPortsCollection[0].AddRef())
        {
            using(SpeechListener aListener =
                aPort.SpeechListener.AddRef())
            {
                string s  = aListener.GetName()
            }
        }
    }
}

void foo (CTAPIApplication TAPIApplication)
{
    using (new LifeTimeScope())
    {
        string s =
            TAPIApplication.Ports.AddRef()[0].AddRef()
            .SpeechListener.AddRef().GetName();
    }
}

The LifeTimeScope class, shown in Figure 6, acts as a helper to call Dispose on all object wrappers that were touched by AddRef within a using block. AddRef got a few changes as well (see Figure 7). While these are all good changes, you still need to make the call to AddRef yourself.

Figure 7 Revised AddRef (Close Figure 7)
public static class ReferenceCountingExtensions
{
    static public System.Collections.Generic.Dictionary<object, object>
        theHashOfReferenceCountedClasses =
        new System.Collections.Generic.Dictionary<object, object>();

    public static T AddRef<T>(this T resource)
    {
        object theObject=null;
        theHashOfReferenceCountedClasses.TryGetValue(resource,
            out theObject);
        if (theObject == null)
        {
            theObject = new RefCounted<T>(resource);
            theHashOfReferenceCountedClasses[resource] = theObject;
        }
        else
        {
            ((RefCounted<T>)theObject).AddRef();
        }
        System.LifeTimeScope.Current
            .theListOfWrapperObjectsToCallDisposeOn.Add(theObject);
        return (T) resource;
    }
}
Figure 6 LifeTimeScope (Close Figure 6)
public  class LifeTimeScope : IDisposable
{
    private static LifeTimeScope _Current = null;

    private static Stack<LifeTimeScope> theScopeStack =
        new Stack<LifeTimeScope>();

    public static LifeTimeScope Current
    {
        get{ return _Current; }
    }

    public System.Collections.ArrayList
        theListOfWrapperObjectsToCallDisposeOn =
        new System.Collections.ArrayList();

    public LifeTimeScope()
    {
        _Current = this;
        theScopeStack.Push(this);
    }

    public void Dispose()
    {
        foreach (IDisposable theObject in
            theListOfWrapperObjectsToCallDisposeOn)
        {
            theObject.Dispose();
        }
        _Current = theScopeStack.Pop();
    }
}

Memory Leaks

The most common issue you will run into is that your application leaks memory due to event subscriptions. For example, you could write a Windows Forms user control that subscribes to the NetworkChange event in its constructor like this:

NetworkChange.NetworkAvailabilityChanged += new
    NetworkAvailabilityChangedEventHandler(ctrl_NetAvailChangedHandler);

The user control and the NetworkAvailabilityChanged static class hold bidirectionally strong references to each other. Because NetworkAvailabilityChanged is a static root reference, the user control will not get released from memory until the event handler is removed.

If you don't unsubscribe an event from an object that exceeds the lifetime of its referenced object—in this case NetworkAvailabilityChanged and its referenced user control—you automatically create a memory leak. This problem is a common one and applies to all sorts of Observer pattern scenarios, including data binding.

The .NET Framework 2.0 documentation says you should unsubscribe the event in the Dispose method of the object. It's also a good idea to make sure Dispose is called. That's because in the .NET Framework 2.0 world events are resources that you need to care about. However, in the .NET Framework 3.0 you can employ a solution based on the WeakEvent pattern.

Finally, another way to assure destruction of unmanaged resources is by using the HandleCollector class, which helps you run GC.Collect occasionally. Figure 8 provides an example Create an instance of HandleCollector by providing three parameters: Handle Name (string), Initial Threshold (int), and Maximum Threshold (int). The initial threshold is the point at which the garbage collector can start performing garbage collection. The maximum threshold is the point at which the garbage collector must perform garbage collection.

Figure 8 Using HandleCollector (Close Figure 8)
// Handle Collector offers an easy way to have garbage collection
// started once we have 30+ active (and possibly destroyed) objects

class MyClassThatForcesGarbageCollectionIf30HandlesAreActive
{
    // Create a new HandleCollector
    static readonly HandleCollector appHandleCollector =
        new HandleCollector("ApplicationUnmanagedHandles", 5, 30);

    public MyClassThatForcesGarbageCollectionIf30HandlesAreActive()
    {
        // Increment Handle Count
        appHandleCollector.Add();
    }
    ~MyClassThatForcesGarbageCollectionIf30HandlesAreActive()
    {
        // Decrement Handle Count
        appHandleCollector.Remove();
    }
}
}

This was a brief overview of lifetime management in .NET or COM classes. I hope you can now recognize and solve these issues. For more information, please visit the links in the "Lifetime Management Resources" sidebar.

Send your questions and comments to clrinout@microsoft.com.

Tim Fischer lives in Germany and is a Program Manager for Visual Studio and Expression products at Microsoft. He holds a degree in computer science from the University of Stuttgart and worked for 12 years as a consultant for model-driven development. He is the founder of tangible engineering, a company dedicated to building world-class .NET software factories.


Basic Instincts: Extension Methods

Basic Instincts
Extension Methods
Adrian Spotty Bowles

Contents
Creating and Using Extension Methods
The Advantage of Extension Methods
Security Concerns
Overloading with Extension Methods
Which Types
Extending Object Type
Extension Methods in .NET Framework 2.0 Apps

W
ith traditional object-oriented development, extending classes is typically achieved by deriving from a base class and enhancing the functionality in the derived class. Visual Basic® still supports these object-oriented concepts, but classes are sometimes marked NotInheritable to prevent modification of their behavior through inheritance. As a result, there is no way to customize these classes. An example of this is the System.String class.

A new feature made available in Visual Basic 2008, however, lets you extend any existing type's functionality, even when a type is not inheritable. And these extension methods play a crucial role in the implementation of LINQ.

Many types that already exist can't be easily updated without breaking existing code. An example of this is the interface IEnumerable(Of T). In order to support LINQ, new methods had to be added to this interface, but changing the interface by adding new methods would break compatibility with existing consumers. Adding a new interface was a possibility, but creating a new interface to supplement the existing IEnumerable(Of T) interface would have appeared to be an odd design.

What was needed was a way to extend existing types with new functionality without changing the existing contract.

Extension methods provide a simple mechanism to extend types in the system (value, reference, and interface types) with new methods. These methods extend the original type and can be called like regular, defined instance methods, but they leave the original type and its methods untouched. Extension methods create the illusion that they are defined on a real type, but, in reality, no changes are made to the original types.

This is not a standard object-oriented concept; it is a specific Microsoft® .NET Framework implementation feature. While this feature opens up a new set of possibilities, it's worth noting that the underlying intermediate language (IL) code generated by the compiler is really doing nothing new or specific to the .NET Framework 3.5. In actuality, it is simply making a shared method call.

This means you have the capability to use this feature in Visual Basic 2008 to target earlier versions of the .NET Framework. It shouldn't introduce any additional security issues since this feature doesn't change the type being extended and doesn't actually do anything that you couldn't already do with earlier versions of the Framework.

Creating and Using Extension Methods

The code in Figure 1 shows an AlternateCase extension method, which returns an alternately cased string. This will extend the System.String class, which, as I noted, is marked NotInheritable.

Figure 1 AlternateCase Extension Method (Close Figure 1)
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim s As String = "Hello World"

'Call AlternateCase method as an extension member on
'variable s
        Console.WriteLine(s.AlternateCase)

'Invoke if it was not an extension method or
'resolve conflicts
Console.WriteLine(Extensions.AlternateCase(s))

    End Sub
End Module

Module Extensions
<Extension()> _
Function AlternateCase(ByVal x As String) As String
        Dim bCase = False
        Dim SB as new System.Text.StringBuilder
        For Each c In x
            If bCase Then
                SB.Append(c.ToUpper)
            Else
                SB.Append(c.ToLower)
            End If
            bCase = Not bCase
        Next
        Return SB.ToString
    End Function
End Module

I'll step you through this example. The method declaration contains the attribute that identifies this method as an extension method. Extension methods are defined by the application of the extension attribute found in the System.Runtime.CompilerServices namespace contained with System.Core.Dll. For those of you not familiar with attributes, they can be used to provide metadata that is used by the compiler to trigger additional functionality. In this case, the extension attribute provides information that tells the compiler that method can be called on the type identified as the first argument to the extension method—String, in my example.

The Visual Basic language insists that all extension method declarations be placed within a Module. And by definition, extension methods can be created as either subroutines or functions, but trying to declare them as anything else, such as properties or events, is prohibited. There are no restrictions on the types that can be extended—you can extend any class, structure, or interface.

Extension methods are restricted to modules in order to ensure that they are easy to consume. This design is intended to build upon the concept of bringing new APIs into scope within a Visual Basic program using the Imports statement to bring all methods in a namespace into scope.

Every method in Visual Basic has to be declared inside a type, whether it is a class, structure, module, or anything else. Extension methods are no exception, though they differ slightly in that they are defined in a type outside of the type they are extending. This opens the potential for naming conflicts when using extension methods from multiple modules or namespaces.

Restricting the declaration of extension methods to modules dramatically simplifies the design since modules can't participate in inheritance chains and there is no concept of partial modules. Using an Imports statement with a Namespace/Module brings those shared methods into scope and allows you to call the methods without having to fully qualify them.

As an example, say I place the Module Extensions within a different namespace, such as this:

Namespace NewFeatures
Module Extensions
<Extension()> Function AlternateCase(ByVal x As String) As String
        Dim bCase = False
        Dim SB as new System.Text.StringBuilder
        For Each c In x
            If bCase Then
                SB.Append(c.ToUpper)
            Else
                SB.Append(c.ToLower)
            End If
            bCase = Not bCase
        Next
        Return SB.ToString
End Module
End Namespace

To bring the extension method declarations within this namespace into scope, I would use the following statement:

Imports NewFeatures
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim s As String = "Hello World"

'Call AlternateCase method as an extension member on 'variable s
        Console.WriteLine(s.AlternateCase)

'Invoking if it was not an extension method or
'resolve conflicts
Console.WriteLine(Extensions.AlternateCase(s))
    End Sub
End Module

Extension methods are not unique to Visual Basic; C# has a similar implementation with extension methods having to be declared in a static class. This provides functionality similar to that of a module. If you work with both C# and Visual Basic, extension methods allow interoperability—you can create extension methods in one language and consume them in the other.

Note that if you've upgraded a project from an earlier version, you'll need to upgrade your Framework target to version 3.5 and then manually add a reference to System.Core.Dll. If you don't want to upgrade your Framework target, you can still take advantage of extension methods by recreating the ExtensionAttribute—more on this later. You'll also need to Import the System.Runtime.CompilerServices namespace (which is where ExtensionAttribute resides) or fully qualify the attribute when you use it.

The final piece of the example is the call. Improved IntelliSense® allows you to see the extension methods along with all the existing instance methods. You'll see that extension methods have a slightly different icon (a blue down arrow, as shown in Figure 2) to differentiate them from instance methods.

Figure 2 Extension and Instance Method Icons
Figure 2 Extension and Instance Method Icons(Click the image for a smaller view)
Figure 2 Extension and Instance Method Icons
Figure 2 Extension and Instance Method Icons (Click the image for a larger view)

The Visual Studio® IDE functionality—such as Goto Definition, Rename, and so on—works for extension methods just as for normal instance methods. You'll notice that tooltips also show the extension attribute when displaying the signature.

You may have noticed other extension methods in IntelliSense, such as First and FirstorDefault. These are some of the extension methods that have already been provided as part of LINQ and are included in the default .NET Framework 3.5 references. They are extending a generic interface System.Collections.Generic.IEnumerable(Of T) from within the System.Linq namespace.

LINQ functionality is enabled by default in version 3.5 projects, which means you get all the extension methods defined in System.Core.Dll. You can confirm this by looking at the default assemblies references and imported namespaces—going to Project Properties | References | Imported Namespaces will show System.Linq already checked. You can also use a file-level Imports statement (Imports System.Linq) to bring these methods back into scope.

The Advantage of Extension Methods

Extension methods allow you to add functionality to a type that you don't want to modify, thus avoiding the risk of breaking code in existing applications. You can extend standard interfaces with additional methods without physically altering the existing class libraries. You can extend .NET types and older COM/ActiveX® control types for new code without risk of breaking old applications that use these types.

Prior to extension methods, in order to add functionality to classes and interfaces, you had a few options:

  • You could add the functionality to the source code, but this requires access to source code that may not be available.
  • You could use inheritance to inherit the functionality contained within one type into a new derived type, but not all types were inheritable.
  • You could re-implement the functionality from scratch.

All three of these approaches made extending third-party types extremely difficult. If you could get the source code and recompile the component, you would also need to ensure that you maintained compatibility with the original component to avoid breaking existing applications that used the component. If you created a new derived class or re-implemented the functionality from scratch, you would need to either fix all existing applications to use the new component or branch between the original functionality and the new updated functionality—and this would require two versions of the component.

None of these solutions were ideal. With extension methods, you can simply add the functionality to existing classes. All existing applications continue to work with the original binaries as nothing has been changed in them, but applications can now extend the functionality going forward.

Performance is relatively unaffected. Even though your Visual Basic source code may look different, it compiles to a simple shared method call and generates the same IL as if you had used the shared method call. So there is no performance hit through using them.

That said, extension methods are not a replacement for object-oriented concepts. If you are familiar with object-oriented concepts and have a well-designed application, you are probably already using inheritance somewhere in your application and should continue doing so. You probably own many of the types you are using, meaning you have access to the source code and can extend the functionality through normal object-oriented concepts. Extension methods really provide a way to enhance classes that you previously could not extend. Extension methods should be used sparingly, when other approaches are not possible or suitable.

Security Concerns

I have often heard the concern that extension methods can be used to hijack or subvert the intended behavior of existing methods. Visual Basic addresses this by ensuring that, wherever possible, an instance method is preferable over an extension method.

The language allows extension methods to be used to create overloads for existing instance methods with different signatures. This allows extension methods to be used to create overloads, while preventing the existing instance method from being overridden. If an extension method exists with the same signature as an instance method, the shadowing rules that are built into the compiler will prefer the instance method, therefore eliminating the possibility of an extension method overriding existing base class instance functionality (see Figure 3).

Figure 3 Extension Method Shadowing (Close Figure 3)
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim x As New ClsFoo
        Dim s As String = x.Method01
    End Sub
End Module

Class ClsFoo
    Function Method01() As String
        Return "Instance"
    End Function
End Class

Module ExtMethods
    <Extension()> _
    Function Method01(ByVal a As clsFoo) As String
        Return "Extension"
    End Function
End Module

As the implementation of extension methods does not change the original type, this additional functionality is known only to those applications that are written to use the extension method. Existing applications that use the original type will see no difference.

If you take a look at the underlying IL that is generated by the compiler, you'll see that the extension method call is transposed into a simple shared method call. Thus, there are no increased security risks from using extension methods since you are doing nothing that you couldn't already do through a normal shared method call.

Overloading with Extension Methods

Extension methods let you overload instance methods. To help ensure that applications work correctly, there are a lot of rules built into the compiler to provide consistent binding behavior. The goal was to make extension methods appear to be instance methods, but also provide a consistent experience—when you enter legacy code into a project using extension methods, the code should continue to work in the same way.

An instance method with an acceptable signature using widening conversion will almost always be preferred over an extension method with an exact signature match. If this results in binding to the instance method when you really want to use the extension method, you can explicitly call the extension method using the shared method calling convention. This is also the way to disambiguate two methods when neither is more specific.

Fields and properties always shadow extension methods by name. Figure 4 shows an extension method and public field with the same name and various calls. Though the extension method contains a second argument, the field shadows the extension method by name and all calls using this name result in accessing the field. The various overloaded calls will all compile, but their results at run time may be unexpected since they will bind to the property and use the default property behavior to return a single character or result in a runtime exception. It is important to choose the names of your extension methods so you avoid clashes with properties, fields, and existing instance methods.

Figure 4 Extension Method and Public Field with Same Name (Close Figure 4)
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim x As New Foo

        'Field - (Test)
        Console.WriteLine(x.TypeField)
        'Default Property Transformation on field resulting in T
        Console.WriteLine(x.TypeField(0))

        'Invalid Cast Exception Converting "fred" to integer
        'to use in default property transformation
        Console.WriteLine(x.TypeField("fred"))
    End Sub
End Module

Class Foo
    Public TypeField As String = "Test"
End Class

Module Extension
    <Extension()> _
    Function TypeField(ByVal a As Foo, ByVal x As String) As String
        Return "Extension"
    End Function
End Module

The rules in the compiler regarding which method the calling code will be bound to are complex. These rules are weighted towards selecting an existing instance method wherever possible. However, there are a few exceptions. The general points to consider are as follows:

  1. Instance methods will almost always be considered before extension methods. Widening instance methods are considered before extension methods. Non-applicable instance method candidates, such as those that are out of scope, are not considered in the overload resolution.
  2. Instance methods with the correct type-matching signature will be considered before methods with widening conversions.
  3. Class Properties/Fields will shadow extension methods by Name.

It should be clear that the name and signature of the extension method are critical in determining the method to be chosen by the compiler, whether it is an extension or instance method.

Which Types

Another consideration relates to which types you are extending. Although you can extend virtually any type, this doesn't mean that you should. You should be aware of "silent rebinding," which can be described as when a method that is being bound to changes, but this change is not immediately obvious to the developer. No warning or error is generated and the only indication that this has occurred is the resulting change in behavior caused by using a different method.

Imagine that you create an extension method that overloads a method in a third-party class library. Your application works as expected. The third-party tool provides an update, which you install on your machine. The application compiles just fine with no errors. But when you run the application, the behavior is incorrect. This may occur if the updated component included additional overloads to the method that you had extended, resulting in the additional instance method overload being called and not your extension method.

Extension methods are fragile in versioning scenarios. This is one reason why you should use extension method names very carefully—especially when used with types that are frequently updated with new overloads. Using Option Strict On can help prevent silent rebinding, as it will alert you to certain conditions where implied conversions can take place.

Extending Object Type

You are able to extend type Object, which allows the method to be seen on any type that derives from type Object. But if you create a type as object and try to consume the extension method on it, you won't see the extension method. This is because extension methods do not support late binding.

Note that in Figure 5, the first call will work because variable s is of type Integer and this derives from Object. As a result, you will see the extension method. Variable T, however, is of type object and is late-bound, and therefore will result in a compile error.

Figure 5 Support for Inheritance (Close Figure 5)
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim s As Integer = 1234
        Console.WriteLine(s.Foo) '<- This will work

        Dim T As Object
        T = 1234
        Console.WriteLine(T.Foo)  '<- This will fail
    End Sub
End Module

Module Extensions01
    <Extension()> _
    Function Foo(ByVal b As Object) As String
        Return "Bar01"
    End Function
End Module

The code could be fixed by changing to the following:

Dim T = 1234
Console.WriteLine(T.Foo)

This will work now because of the type inference feature. Variable T gets inferred as integer based on the value being assigned to it. This differs from Visual Studio 2005, where T would have been type Object containing a boxed integer.

Extension Methods in .NET Framework 2.0 Apps

As I noted earlier, extension methods are not doing anything in the underlying IL code that could not be done using Visual Basic 2005; they are simply making a shared method call. What has changed is the compiler's ability to identify specific methods and allow the change in the calling method syntax, which gives the appearance that the type being used has new functionality and makes the extensions work just like the existing instance methods.

Another new feature in Visual Basic 2008 is multi-targeting, which lets you use Visual Basic 2008 to write code that targets a specific version of the Framework. Since the same compiler is used for multi-targeting, it is possible to use extension methods for applications targeted, for example, to version 2.0 of the Framework. This does take a little workaround, as the IDE prevents adding the required references to a .NET Framework 2.0 targeted application.

If you create a 3.5 application and change the target Framework to version 2.0 (using Project Properties | Compiler | Advanced | Target Framework), you'll see some of the references highlighted as missing, including those containing most of the out-of-the-box LINQ functionality. Trying to add these references to a 2.0 project results in a dialog informing you that they require a different version of the targeted Framework (see Figure 6).

Figure 6 References for Changed Target to the .NET Framework 2.0
Figure 6 References for Changed Target to the .NET Framework 2.0(Click the image for a smaller view)
Figure 6 References for Changed Target to the .NET Framework 2.0
Figure 6 References for Changed Target to the .NET Framework 2.0 (Click the image for a larger view)

Extension methods can be created and used for 2.0 applications after you have created your own System.Runtime.CompilerServices.Extension attribute. You must first create a new project targeting version 2.0. The default references will not include System.Core, which contains the extension attribute. In the Applications Project Properties, go to the Application Tab, clear the Root Namespace, and then create your own extension attribute in the correct System.Runtime.CompilerServices namespace. The code in Figure 7 shows the creation of the extension attribute and then using this to declare and use extension methods. Once you compile the application, you can call the extension methods.

Figure 7 Creating an ExtensionAttribute Class (Close Figure 7)
Imports System.Runtime.CompilerServices

Module Module1
    Sub Main()
        Dim i_x As Integer = 1
        Dim i_s As String = "Test"

        Dim x = i_x.IntegerExtension
        MsgBox(x)

        Dim y = i_s.StringExtension
        MsgBox(y.ToString)
    End Sub
End Module

'CREATE A USER-DEFINED ATTRIBUTE CALLED EXTENSION IN THE CORRECT NS
Namespace System.Runtime.CompilerServices
    <AttributeUsage(AttributeTargets.Assembly Or AttributeTargets.Class Or AttributeTargets.Method, AllowMultiple:=False, Inherited:=False)> Class ExtensionAttribute
        Inherits Attribute
    End Class
End Namespace

'DEFINED EXTENSION METHODS WITH USER-DEFINED ATTRIBUTE ON THEM
Module ExtMethods
    <Extension()> _
Function StringExtension(ByVal a As String) As String
        Return "Success"
    End Function

    <Extension()> _
Function IntegerExtension(ByVal a As Integer) As Integer
        Return 100
    End Function
End Module

Here are some important considerations regarding 2.0-targeted applications. First, type inference is off by default. To turn it back on, use Option Infer On. You'll probably want to do when you migrate .NET Framework 2.0 applications to the version 3.5.

Second, to use extension methods, you need to create your own extension attribute that mimics the one in System.Core. The IDE will prevent you from using the System.Core reference on a 2.0-targeted project.

Finally, this technique will not work for ASP.NET applications targeting the .NET Framework 2.0 because they have a runtime dependency on the 2.0 command-line compiler. When those apps are deployed to Web servers that only have the .NET Framework 2.0 installed, they will not work because the 2.0 VBC.EXE command-line compiler does not understand extension methods.

Even if you don't want to start writing applications for the .NET Framework 3.5 or you have existing applications in version 2.0 that you have to maintain, you can still benefit from upgrading to Visual Basic 2008. The extension method functionality is built into the compiler and can be used for any version of the Framework that the Visual Basic 2008 compiler is able to target.

Send your comments and questions to instinct@microsoft.com.

Adrian Spotty Bowles has developed using every version of Visual Basic and has managed to find his way to Redmond, WA, where he works on the Visual Basic Product team as a Software Design Engineer Tester focused on the Visual Basic compiler. During the Visual Basic 2008 release, he worked on many of the language features, including Extension Methods. You can reach Spotty at Abowles@microsoft.com.


Office Space: Events in SharePoint 2007

Office Space
Events in SharePoint 2007
Ted Pattison

Code download available at: OfficeSpace2007_11.exe (167KB)
Contents
Event Receiver Classes
Before Events and After Events
Validating Before Events
After Event Handlers
Binding Handlers to Events
Using the WSS Object Model

W
indows® SharePoint® Services (WSS) 3.0 offers developers many significant improvements for building custom business solutions based on SharePoint sites. One of the biggest developer-focused enhancements has to do with a new infrastructure for handling server-side events. For example, when a user performs an action that modifies content in a SharePoint site—say, uploading a new document or deleting an item from a list—there is an opportunity for a developer to respond to this user action with an event handler that executes server-side logic written in C# or Visual Basic®.

While WSS 2.0 also provided an infrastructure for server-side event handling, things are vastly improved in WSS 3.0. Figure 1 shows an overview of the differences between the event-handling support in WSS 2.0 versus WSS 3.0. Furthermore, it's important for you to understand that the new WSS 3.0 event-handling infrastructure is not a mere refinement of what was already there in the previous version. The new WSS 3.0 event-handling infrastructure represents a complete replacement for the WSS 2.0 event-handling infrastructure. One of the biggest changes is that event handling is implemented at the list level and not the document library level. That means events that fire on list definitions and list items work on all types of lists, including document libraries.

Figure 1 Event-Handling in WSS 2.0 and WSS 3.0 (Close Figure 1)

Event Type | WSS 2.0 Infrastructure | WSS 3.0 Infrastructure
Events on document libraries | Supported | Supported
Multiple event handlers bound to a single event | Not Supported | Supported
Events on lists | Not Supported | Supported
Events on list definitions | Not Supported | Supported
Events on sites | Not Supported | Supported
Before-the-fact, synchronous event handlers with cancellation | Not Supported | Supported
After-the-fact, asynchronous event handlers | Supported | Supported
Event handler classes | Must implement the IListEventSink interface | Must inherit from receiver base classes in WSS object model

The older WSS 2.0 infrastructure has a completely different programming model and requires a different approach for wiring event handlers to events. WSS 3.0 provides support for both the new version 3.0 event-handling infrastructure as well as the older infrastructure from WSS 2.0. You should understand, however, that the support in WSS 3.0 for the older version 2.0-style of event handlers has only been included to provide backward compatibility when integrating code written for WSS 2.0 solutions into a WSS 3.0 environment. When you begin developing solutions for WSS 3.0 and Microsoft® Office SharePoint Server (MOSS) 2007, you should employ the new WSS 3.0 event-handling infrastructure, which is based on new concepts and the programming model that I am going to introduce in this month's Office Space column.

Event Receiver Classes

Event handling in WSS 3.0 is based on event receiver classes. More specifically, you create a new event receiver class by inheriting from one of several event receiver base classes that are provided as public classes within the WSS object model. To do this, you should create a new class library project that references the core WSS assembly named Microsoft.SharePoint.dll.

The list of the WSS receiver base classes available through the WSS object model is shown in Figure 2. Once you create a class definition that inherits from one of these WSS receiver base classes, you add event handlers by overriding selected methods.

Figure 2 WSS Event Receiver Base Classes (Close Figure 2)

Class | Description
SPListEventReceiver | Provides event handling when users add, change, and remove columns for a list definition.
SPItemEventReceiver | Provides event handling when users modify items within a list or when users modify documents within a document library.
SPWebEventReceiver | Provides event handling when users move or delete a site collection or a site.
SPEmailEventReceiver | Provides event handling when users send e-mail to an e-mail- enabled list.

Once you have implemented a custom receiver class with one or more event handlers, you must then compile it into an assembly with a strong name and install it in the global assembly cache (GAC). You can then bind your event handlers to events exposed by a host object such as a site, a list, or a document library. As you will see later in this column, you have two choices when it comes to binding event handlers. You can bind event handlers using a Feature with declarative XML syntax or you can bind them programmatically using the WSS object model. Once you have bound an event-handler method to a specific event, WSS then takes care of executing your handler in response to the user actions that trigger the event.

Before Events and After Events

WSS 3.0 events can be separated into two main categories: Before events and After events. Before events fire in response to a user's action at a time before WSS has written any data back to the content database. A key point is that Before events occur early enough in the request processing lifecycle that they support the cancellation of the user's action. Therefore, Before events provide a great opportunity to perform security checks and custom validation.

WSS also supports After events, whose event handlers execute after WSS 3.0 has written back to the content database to commit the user's action. After events do not support canceling the user's action. Instead, After events are used to kick off custom processing logic designed to run whenever content has been modified.

The event handler for a Before event runs on the same thread that is processing the user's action (such as adding an item). Therefore, Before events have a blocking nature and are commonly referred to as synchronous events. WSS executes the event handlers for After events on a separate thread so their processing does not block the response that is sent back to the user. For this reason, After events are commonly referred to as asynchronous events. Notice also that Before events are based on overridable methods whose names end with "ing" (for example, FieldDeleting), while After event method names end with "ed" (for example, FieldDeleted).

Let's look at an example of authoring a new custom receiver class that defines an event handler for a Before event and for an After event. Figure 3 shows a custom receiver class that inherits from the SPListEventReceiver class and overrides the FieldDeleting and FieldDeleted methods. You should take note that when WSS calls one of these event handlers, it passes a single parameter of type SPListEventProperties named properties.

Figure 3 Custom Receiver Class (Close Figure 3)
Imports Microsoft.SharePoint

Public Class MyListEvents
    Inherits SPListEventReceiver

    Public Overrides Sub FieldDeleting ( _
        ByVal properties As SPListEventProperties)
    '*** your custom event handling code goes here
    End Sub

    Public Overrides Sub FieldDeleted ( _
        ByVal properties As SPListEventProperties)
    '*** your custom event handling code goes here
    End Sub

End Class

Since the FieldDeleting method handles a Before event, it supports canceling the user's action. Imagine a scenario where you want to prohibit all users including the site collection administrator from being able to delete columns from a particular list definition. You can accomplish this by assigning a value of True to the Cancel property of the SPListEventProperties parameter named properties. When canceling the user's action, you should also assign a custom error message by assigning a string value to the ErrorMessage property. This error message will be shown directly to the user:

Public Overrides Sub FieldDeleting( _
    ByVal properties As SPListEventProperties)
    '*** cancel user action
    properties.ErrorMessage = "Deleting not supported!"
    properties.Cancel = True
End Sub

When an event handler for a Before event assigns a value of True to the Cancel property, WSS responds by short-circuiting the request and canceling the user's action. Instead of deleting the column as the user requested, WSS displays the custom error message to the user. The key point here is that Before events provide you with a layer of defense against unwanted modifications.

Now that you have seen the basics of canceling the user's action in the event handler for a Before event, let's create another event handler for a Before event with some conditional logic. Imagine you have a requirement where you need to run a security check to ensure users meet certain criteria before allowing them to delete items from a list or to delete documents from a document library. Have a look at the ItemDeleting event handler in the receiver class definition shown in Figure 4. Notice that the conditional logic cancels the user's action when it determines the user does not possess the permissions of a site administrator.

Figure 4 Simple Security Check (Close Figure 4)
Public Class MyItemEvents
    Inherits SPItemEventReceiver

    Public Overrides Sub ItemDeleting( _
        ByVal properties As SPItemEventProperties)

        If (Not properties.OpenWeb().CurrentUser.IsSiteAdmin()) Then
            Dim ErrorMessage As String
            ErrorMessage = "Only site administrator can delete items"
            properties.ErrorMessage = ErrorMessage
            properties.Cancel = True
        End If

    End Sub

End Class

As you can see, this event-handler logic requires that the current user has certain permissions before permitting a deletion to occur. Taking this example one step further, you can write an event handler to determine whether users can delete items depending on whether they are members of an Active Directory® group. Since WSS is built on top of the ASP.NET runtime, you can accomplish this with a simple ASP.NET programming technique and query the IsInRole method exposed by the User property of the HttpContext object:

If (Not HttpContext.Current.User.IsInRole( _
    "MyDomain\ContentManagers")) Then
    Dim ErrorMessage As String
    ErrorMessage = _
    "Only members of ContentManagers can delete items"
    properties.ErrorMessage = ErrorMessage
    properties.Cancel = True
End If

Validating Before Events

Now let's move on and examine the possibilities of writing validation logic in an event handler for a Before event. This makes it possible to write event handlers that make users conform to application-specific constraints when they add or modify items within a list. For example, you can validate that the ship date for an invoice is not a Saturday or Sunday or that a phone number or e-mail address meets a pattern defined by a regular expression.

One of the most powerful aspects of writing validation logic for Before events is that you can perform a validation that spans columns. In the sample code for this month, I use a Sales Leads list that contains a column for a work phone number and a column for an e-mail address. I have an event handler that will perform a validation on each item with the Sales Leads list to require that either the work phone number or the e-mail address has a non-blank value. The basic idea here is that sales leads are not valid if someone has not entered in any contact information.

There are two ways for a user to modify a list item. A user can add a new item or update an existing item. When writing validation logic for Before events, you must typically add redundant code behind the event handlers bound to the ItemAdding event and the ItemUpdating event. Therefore, it's best to maintain the validation logic in a separate method that can be called from two different event handlers. The SalesLeadIsValid method shown in Figure 5 accepts two string parameters named Phone and E-mail and returns True if either of these parameters is not blank.

Figure 5 Validating Input (Close Figure 5)
Private Function SalesLeadIsValid(
    ByVal Phone As String, ByVal Email As String) _
    As Boolean

    '*** make sure we have either phone or e-mail
    If (String.IsNullOrEmpty(Phone) And _
        String.IsNullOrEmpty(Email)) Then
        Return False
    Else
        Return True
    End If

End Function

With the validation logic inside the SalesLeadIsValid method, you can call this method from event handlers bound to the ItemAdding event as well as the ItemUpdating event. Note that within an event handler for an ItemAdding event or a ItemUpdating event, you must retrieve the column values against which you want to perform validations. The SPItemEventProperties parameter exposes two collections named BeforeProperties and AfterProperties. The idea is that you can retrieve the value of any column in a state either before or after the user changed it.

Figure 6 shows an example of implementing an ItemAdding event handler that retrieves the After values for both the Phone and E-mail and passes them to the SalesLeadIsValid method to determine whether to allow the user's action to succeed.

Figure 6 Validating Against an Action (Close Figure 6)
Public Overrides Sub ItemAdding( _
    ByVal properties As SPItemEventProperties)

    Dim Phone As String, Email As String
    Phone = _
        CType(properties.AfterProperties("WorkPhone"), _
        String)
    Email = _
        CType(properties.AfterProperties("EMail"), _
        String)

    If (Not SalesLeadIsValid(Phone, Email)) Then
        Dim ErrorMessage As String
        ErrorMessage = _
            "Sales lead must have valid phone or email"
        properties.ErrorMessage = ErrorMessage
        properties.Cancel = True
    End If
End Sub

Take a closer look at how the event handler retrieves the two column values from the AfterProperties collection. AfterProperties is a collection with a default parameterized property (an indexer) that accepts the name of a column. Note that the parameter passed to AfterProperties is a case-sensitive string value. You should also understand that this parameter represents the physical name of the column as opposed to the display name.

After Event Handlers

An After event is different from a Before event because it fires after an action has been completed and it does not provide the developer with an opportunity to cancel the action. After events provide the developer with an opportunity to kick off custom processing that should execute when the user has successfully completed an action that has modified content within a SharePoint site. For example, once a user has added a new item to a list, an After event can be used to reformat one of the columns to maintain data integrity or to send out a notification e-mail message to a specific user or distribution list.

Let's look at an example of writing an event handler for our Sales Lead list that will be bound to the ItemAdded event. In this example, there is a requirement to format all sales lead names with all capital letters. An event handler takes the column value with the name of the sales lead and reformats it using uppercase text.

Note that you have to be careful when you write an event handler for an After event and you modify the item on which the event is based. The problem is that modifying the item will cause the event to fire again. When the event fires again it wants to modify the item, starting a recursive loop. To avoid this you must disable the event firing on the current item while you are modifying it. This is done by calling the DisableEventFiring method supplied by the underlying base class. After you have modified the item, you should then call EnableEventFiring. Here's how all the pieces fit together:

Public Overrides Sub ItemUpdated( _
    ByVal properties As SPItemEventProperties)

    Me.DisableEventFiring()
    Dim SalesLead As String = _
        CType(properties.ListItem("Title"), String)
    '*** reformat item column values as required
    properties.ListItem("Title") = SalesLead.ToUpper()
    properties.ListItem.Update()
    Me.EnableEventFiring()
End Sub

Now you have seen several different examples of writing event handlers for both Before events and After events. You should now be more comfortable with why and how to write event handlers. The main question that remains is how to bind them to events on host object within a SharePoint site, so let's look at that now.

Binding Handlers to Events

Earlier I mentioned that receiver classes must be compiled into an assembly DLL with a strong name and installed in the GAC. Your options for binding event handlers are to use declarative XML in a Feature or programmatically bind the two via the WSS object model. However, both approaches still require that you know the complete four-part assembly name, as well as the namespace-qualified name, of the receiver class containing the event handlers to which you want to bind.

I will begin by discussing how to bind event handlers to a list type using declarative XML within a Feature. You can accomplish this by adding a Receivers element with one or more inner Receiver elements. In the sample code that accompanies this column, there is a Feature with a Receivers element that binds two different event handlers to the WSS Contacts list type (see Figure 7). The Contacts list has a list type identifier of 105. This means that any list that is created from the Contacts list inside a site that has this Feature activated will automatically have these event handler bindings.

Figure 7 Contacts List Receiver (Close Figure 7)
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">

  <!-- Receivers element used only in feature where Scope=Web -->
  <Receivers ListTemplateId="105" >

    <Receiver>
      <Name>Field_Deleting_Event</Name>
      <Type>FieldDeleting</Type>
      <Assembly>OfficeSpaceEventDemo, [4-part assembly name]</Assembly>
      <Class>OfficeSpaceEventDemo.MyListEvents</Class>
      <SequenceNumber>1000</SequenceNumber>
      <Data>My Custom Data</Data>
    </Receiver>

    <Receiver>
      <Name>Field_Deleted_Event</Name>
      <Type>FieldDeleted</Type>
      <Assembly>OfficeSpaceEventDemo, [4-part assembly name]</Assembly>
      <Class>OfficeSpaceEventDemo.MyListEvents</Class>
      <SequenceNumber>1000</SequenceNumber>
      <Data>My Custom Data</Data>
    </Receiver>

  </Receivers>
</Elements>

Note that inside the Receivers element you must supply a separate Receiver element for each event handler you want to bind. There is no supported technique to bind multiple event handlers at once with a single Receiver element.

There are two mandatory pieces of binding information in a Receiver element. They are the Assembly element and the Class element, which must contain the complete four-part assembly name and namespace-qualified receiver class name respectively. The Name element within a Receiver element can be used to assign a friendly name to the event handler for tracking purposes, but is not required.

The SequenceNumber element within a Receiver element is useful when you have multiple event handlers bound to a single event. The event handler with the lower sequence number always fires before an event handler with a higher sequence number. Therefore, you can add in multiple event handlers and be guaranteed a deterministic execution order.

The Data element within a Receiver element allows you to configure a particular event handler binding with any custom string data you would like. You can access this string value from within the Data element using the ReceiverData property of the SPListEventProperties parameter and use it in an application-specific manner to provide numbers or regular expressions for validation routines or custom error messages:

Public Overrides Sub FieldDeleting( _
    ByVal properties As SPListEventProperties)

    Dim BindingData As String
    BindingData = properties.ReceiverData
  '*** use binding data in application-specific fashion
End Sub

Using the WSS Object Model

While binding event handlers to events declaratively using a Receivers element in a Feature works in some scenarios, it lacks the flexibility you need in many others. For example, the Receivers element can only be used to bind an event handler to a list type on a site-wide basis. This declarative approach does not provide a way to bind an event handler to a single list instance in a site without also binding it to every other list of the same type.

When designing custom SharePoint solutions that utilize event handlers, you will find that binding event handlers with code provides much more flexibility. For example, you can bind an event handler to an event just for a single list.

Let's look at an example of binding the event handler for the ItemAdding event to the Sales Leads list:

Dim SalesLeadsList As SPList = Me.Web.Lists("Sales Leads")
    '*** bind event handler to ItemAdding event
    SalesLeadsList.EventReceivers.Add( _
        SPEventReceiverType.ItemAdding, _
        GetType(MyItemEvents).Assembly.FullName, _
        GetType(MyItemEvents).FullName)

Each SPList object exposes an EventReceivers collection. The EventReceivers collection exposes an Add method with three different overloaded implementations. The Add method in this last example represents the simplest approach, which requires passing three parameters. When you want to bind an event handler through code, you must minimally supply the name of the assembly, the namespace-qualified class name, and the event type using the SPEventReceiverType enumeration from the WSS object model.

Note that while you can embed literal string values inside your code containing the four-part assembly name and namespace-qualified receiver class name, you can also use the code shown earlier to discover these values dynamically.

Calling the Add method represents the easiest approach, but it is not flexible enough for all scenarios. Figure 8 illustrates a more complicated approach in which the event-handler binding is performed by programming directly against a SPEventReceiverDefinition object. This gives you an opportunity to give the binding a specific GUID identifier so you can add and remove them more easily. It also gives you the ability to initialize the SequenceNumber property and the Data property.

Figure 8 Binding Against a GUID (Close Figure 8)
Dim SalesLeadsList As SPList = Me.Web.Lists("Sales Leads")

Dim receivers As SPEventReceiverDefinitionCollection
receivers = SalesLeadsList.EventReceivers
Dim ItemAddingDefId As Guid
ItemAddingDefId = New Guid("D6C452E5-95D4-45e8-934C-30F785867000")
If (Not receivers.EventReceiverDefinitionExist(ItemAddingDefId)) Then
    Dim ItemAddingDef As SPEventReceiverDefinition
    ItemAddingDef = receivers.Add(ItemAddingDefId)
    ItemAddingDef.Name = "ItemAddingHandler"
    ItemAddingDef.Type = SPEventReceiverType.ItemAdding
    ItemAddingDef.Assembly = GetType(MyItemEvents).Assembly.FullName
    ItemAddingDef.Class = GetType(MyItemEvents).FullName
    ItemAddingDef.Data = "My Custom Data"
    ItemAddingDef.SequenceNumber = 1001
    ItemAddingDef.Update()
End If

I have really just scratched the surface of what is possible when binding event handlers through the WSS object model. I demonstrated working with the EventReceivers collection of a list, but you should know that several other types in the WSS object model have an EventReceivers collection and can act as event hosts. Examples of such types include sites, content types, and files.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. Ted has just completed his book titled Inside Windows SharePoint Services 3.0 for Microsoft Press. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group (www.TedPattison.net).


Cutting Edge: Managing the User Experience in AJAX

Cutting Edge
Managing the User Experience in AJAX
Dino Esposito

Code download available at: CuttingEdge2007_11.exe (183KB)
Contents
AJAX and the User Interface
Disabling UI via Element IDs
Disabling UI Elements in Naming Containers
Using the UpdatePanelAnimation Extender
Disabling the Whole Page
One AJAX Postback at a Time
High-Priority Calls

W
e are all used to the stop-and-go nature of the Web. For every request you make by clicking a button or a link, the browser places a request and waits for the response. You work a bit with that page, then click and wait again.

As I've mentioned in recent columns, I am very excited about the possibilities offered by the AJAX paradigm and its related set of technologies. Developers and architects can build a new generation of Web sites in which the stop-and-go pattern is replaced by a more interactive structure. I always enjoy seeing the sincere enthusiasm users have when they first experience the continuous feel of an AJAX application.

In recent columns I have extolled the power of AJAX, listing its many direct and indirect benefits. This month, I want to focus on some issues directly related to implementing the AJAX paradigm by using ASP.NET AJAX Extensions.

Initially built on top of ASP.NET 2.0, ASP.NET AJAX Extensions has been fully integrated into the ASP.NET platform with the release of ASP.NET 3.5. In fact, ASP.NET AJAX sports two programming models for you to choose from—partial rendering and script services. For a deeper discussion of the two, you might want to check out my columns from the September 2007 and October 2007 issues of MSDN® Magazine.

An AJAX application has the power to place Web requests directly from script code, bypassing the browser. When you use the browser to request a page—for example, through a standard Submit button or hyperlink—just one request is processed at a time for each active browser window. Requesting a page via script doesn't have this limitation and multiple requests can take place concurrently. In the July 2007 installment of Cutting Edge, I took advantage of this feature to implement a live progress bar that reports to the client the real progress made by a server-side running task. The possibility of placing concurrent calls from a single browser window opens up a whole new world of possibilities for addressing various issues—including many related to the UI.

In my column this month, I'll review problems, solutions, and necessary tools to implement an effective user interface for Web applications based on ASP.NET AJAX. In particular, I'll focus on applications that use the partial rendering model.

AJAX and the User Interface

A Web page remains active even after the user has placed an AJAX request. This means that the user can click on active elements and start new operations that could interfere with the current request. As with desktop applications, therefore, developers sometimes need to consider temporarily disabling portions of the UI.

How can you disable client page elements for the duration of a server operation? There are a few ways to do this. You can retrieve the ID of the corresponding Document Object Model (DOM) element and enable or disable its state—this is a client-side operation that you accomplish via script using the DOM or perhaps CSS styles. In addition, ASP.NET controls feature the Visible property. When set to false, the Visible property prevents the server control from adding its own markup to the client response. As a result, the expected markup for that control doesn't show up in the client page.

To disable and enable a portion of the user interface in ASP.NET AJAX, you first need to identify the IDs of all the HTML elements that form that portion, as shown in the following code:

// Get the reference to the DOM element
var button1 = $get("Button1");

// Disable the status of the element
button1.disabled = true;

To disable elements, you retrieve the ID and set the disabled property to the appropriate value. Ideally, you should turn off the specific portion of the user interface immediately before placing the request and re-enable that portion immediately after the request has been completed. As for the implementation, this will vary depending on the programming model you have chosen.

If you are using partial rendering, use the eventing model of the PageRequestManager object—this class is the client-side brains behind partial rendering. The full class name is Sys.WebForms.PageRequestManager and its source code is defined in MicrosoftAjaxWebForms.js. PageRequestManager contains all the logic required to set up and control AJAX postbacks. The class is a singleton—which means that it guarantees that exactly one request is pending at a time. You obtain the reference to the unique instance of the page request manager through the following code:

var manager = Sys.WebForms.PageRequestManager.getInstance();

PageRequestManager exposes a few client-side events that let you intercept key steps in the processing of an AJAX postback, as detailed in Figure 1. The occurrence of the BeginRequest event is usually a good time to disable UI elements, and the EndRequest event is the right time to re-enable HTML elements.

Figure 1 Events Exposed by the PageRequestManager (Close Figure 1)

Event | Argument | Description
initializeRequest | InitializeRequestEventArgs | Fired immediately after detecting that a new AJAX request has been issued.
beginRequest | BeginRequestEventArgs | Fired just before the request is sent out.
pageLoading | PageLoadingEventArgs | Fired once the response has been fully downloaded and processed, but before any content on the page is updated.
pageLoaded | PageLoadedEventArgs | Fired once all contents on the page are refreshed.
endRequest | EndRequestEventArgs | Fired once all work is done.

If you opted for a script services approach, then you probably have a JavaScript-powered front end that talks to a server-side façade made of ASP.NET AJAX script Web services. Each call to services is conducted via an explicit asynchronous script call. The developer is in complete control over the start of the remote call and can safely disable any pieces of the UI. Because each call is going to be asynchronous, a callback can be specified for both success and failure of the call. From the callback, then, you can reset the UI back to proper status.

All that said, there are some issues relative to disabling and re-enabling UI elements. I'll delve into them with some scenarios.

Disabling UI via Element IDs

Figure 2 shows the code used to disable some elements of a sample AJAX page (using partial rendering) during postbacks. When the page first loads, handlers for BeginRequest and EndRequest events are registered. The pageLoad function is a conventionally named handler for the application load event fired by the AJAX client library. The pageLoad function receives two arguments—one is of type Sys.Application and the other one is an ApplicationLoadEventArgs object.

Figure 2 Page that Disables UI during Postbacks (Close Figure 2)
<html>
<body>

<script type="text/javascript">
function pageLoad()
{
    var manager = Sys.WebForms.PageRequestManager.getInstance();
    manager.add_beginRequest(OnBeginRequest);
    manager.add_endRequest(OnEndRequest);
}
var lcPostbackElementID;
functionw OnBeginRequest(sender, args)
{
    lcPostbackElementID = args.get_postBackElement().id.toLowerCase();
    if (lcPostbackElementID === "button1")
    {
        $get("Button1").disabled = true;
    }
}

function OnEndRequest(sender, args)
{
    if (lcPostbackElementID === "button1")
    {
        $get("Button1").disabled = false;
    }
}
</script>

<form id="form1" runat="server">

    <asp:ScriptManager ID="ScriptManager1" runat="server" />

    <div id="pageContent">

        <b>Enter some data:</b><br />
        <asp:TextBox ID="TextBox1" runat="server" />
        <asp:Button ID="Button1" runat="server" Text="Post ..."
            onclick="Button1_Click" />
        <br /><br />

        <asp:UpdateProgress runat="server" ID="UpdateProgress1">
            <ProgressTemplate>
                <img src="images/loading.gif" />
            </ProgressTemplate>
        </asp:UpdateProgress>

        <asp:UpdatePanel ID="UpdatePanel2" runat="server"
            UpdateMode="Conditional">
            <ContentTemplate>
                <asp:Label runat="server" ID="Label1" />
            </ContentTemplate>
            <Triggers>
                <asp:AsyncPostBackTrigger ControlID="Button1" />
            </Triggers>
        </asp:UpdatePanel>
    </div>

</form>
</body>
</html>

BeginRequest (and all other events listed in Figure 1) occurs for each AJAX postback raised from within the page. Such events merely notify developers that a request is being placed or has just completed. These events are not like, for instance, the click event of a client or server button. You know that a postback is going to happen, but you don't immediately know why or what UI element is responsible for it. But in order to disable the right portion of the user interface, you need to know exactly which element was clicked.

The data structure that accompanies the BeginRequest event contains a reference to the DOM element that caused the postback. The member is named postBackElement and is accessed like so:

var lcPostbackElement = args.get_postBackElement();

You can then use this information to check the element the user clicked and decide which elements need to be disabled:

if (lcPostbackElement.id.toLowerCase() === "button1")
{
    $get("Button1").disabled = true;
    ...
}

Note that the same information is not associated with the EndRequest event when you restore the original UI. Therefore, you need to cache the reference, or just the ID of the element, to check it again from within the EndRequest event handler.

To disable and enable a DOM element, you need to retrieve a DOM reference to the element and then act on the Boolean disabled property. In ASP.NET AJAX, the $get function is equivalent to the DOM document.getElementById call. More precisely, $get is equivalent to a parameterized call of getElementById that also accepts an optional reference to indicate the root of the DOM subtree to be searched.

It is very simple to temporarily disable UI elements in a script service call. The only thing that changes is the location from which you use the $get function. You disable before making the call and re-enable in the success and failure callbacks, as shown in Figure 3.

Figure 3 Disabling UI Elements in a Script Service Call (Close Figure 3)
function startCall()
{
   $get("Button1").disabled = true;
   ...
   Samples.RemoteService.GetTime(param1, param2,
      onCompleted, onFailed);
}
function onCompleted(results)
{
   // Process response
   ...
   $get("Button1").disabled = false;
   ...
}
function onFailed(error)
{
   $get("Button1").disabled = false;
   ...
}

Disabling UI Elements in Naming Containers

The code discussed so far works just fine, but it is far from perfect. What if you need to disable UI elements embedded in a Master Page or a user control?

Many ASP.NET server controls use a naming container to create a unique namespace for the ID property values of their child controls. This is especially important in templated and iterative data-bound controls, such as GridView and Repeater. For each control, the naming container is the nearest parent control that implements the INamingContainer interface. For example, the UserControl class acts as a naming container for all of its contained controls. The same happens to any control on a page that is associated with a Master Page.

For any ASP.NET control under the jurisdiction of a naming container, its client-side ID is changed to be prefixed with the ID of the parent. Consider the following code snippet:

<form runat="server">
  <asp:Button runat="server" ID="Button1" />
  ...
</form>

The client ID of the button matches the server ID and is always Button1. However, this programming aspect changes suddenly if you move the button in a user control or derive its content page from a Master Page. Consider the following user control:

<%@ Control Language="C#" CodeFile="test.ascx.cs" Inherits="test" %>
<asp:Button runat="server" ID="Button2" Text="Click" />

After embedding the user control in an ASP.NET page, the emitted HTML refers to Button2:

<input type="submit"
       name="UserControl1$Button2"
       value="Click"
       id="UserControl1_Button2" />

The ID of each contained control is now prefixed with the ID of the parent. This behavior is essentially transparent to the developer as long as you limit yourself to server-side programming. If, however, you need to script Button2 from the client, you need to know about naming containers and how they change the client ID of HTML elements in the page DOM.

In Figure 4, you see the same page as that shown in Figure 2, but embedded as a content page in a Master Page. The real client ID of Button1 is now:

Figure 4 Content Page that Disables UI during Postbacks (Close Figure 4)
<%@ Page Language="C#" AutoEventWireup="true"
    CodeFile="DisableUIEmb.aspx.cs"
    MasterPageFile="~/Sample.master" Title="Disable UI Elements (Embedded)"
    Inherits="Samples_DisableUIEmb" Theme="ProgAjax1" %>

<asp:Content runat="server" ContentPlaceHolderID="ScriptPlaceHolder">
    <script type="text/javascript">
    function pageLoad()
    {
        var manager = Sys.WebForms.PageRequestManager.getInstance();
        manager.add_beginRequest(OnBeginRequest);
        manager.add_endRequest(OnEndRequest);
    }
    var lcPostbackElementID;
    function OnBeginRequest(sender, args)
    {
        lcPostbackElementID = args.get_postBackElement().id.toLowerCase();
        if (lcPostbackElementID === "<%= Button1.ClientID.ToLower() %>")
        {
            $get("<%= Button1.ClientID %>").disabled = true;
        }
    }

    function OnEndRequest(sender, args)
    {
        if (lcPostbackElementID === "<%= Button1.ClientID.ToLower() %>")
        {
            $get("<%= Button1.ClientID %>").disabled = false;
        }
    }
    </script>
</asp:Content>

<asp:Content runat="server" ContentPlaceHolderID="ContentPlaceHolder1">

    <div id="pageContent">

        <b>Enter some data:</b><br />
        <asp:TextBox ID="TextBox1" runat="server" />
        <asp:Button ID="Button1" runat="server" Text="Post ..."
            onclick="Button1_Click" />
        <br /><br />

        <asp:UpdateProgress runat="server" ID="UpdateProgress1">
            <ProgressTemplate>
                <img src="images/loading.gif" />
            </ProgressTemplate>
        </asp:UpdateProgress>

        <asp:UpdatePanel ID="UpdatePanel2" runat="server"
            UpdateMode="Conditional">
            <ContentTemplate>
                <asp:Label runat="server" ID="Label1" />
            </ContentTemplate>
            <Triggers>
                <asp:AsyncPostBackTrigger ControlID="Button1" />
            </Triggers>
        </asp:UpdatePanel>
    </div>

</asp:Content>

ctl00_ContentPlaceHolder1_Button1

To properly disable Button1 (or, in general, to script it), you must know this new ID. The exact ID is known only at run time, even though the name can be figured out and hardcoded in an external JavaScript file. Figure 4 demonstrates how you can use ASP-style code blocks and the server-side ClientID property to emit in the browser parametric script code that works around naming containers. Figure 5 shows the sample page in action.

Figure 5 Content Page Using Script to Adjust UI
Figure 5 Content Page Using Script to Adjust UI(Click the image for a smaller view)
Figure 5 Content Page Using Script to Adjust UI
Figure 5 Content Page Using Script to Adjust UI (Click the image for a larger view)

Using the UpdatePanelAnimation Extender

The portion of the UI being disabled is postback-specific and you usually define that using plain script. In most cases, all you need to do is gray out elements; however, you could do even more. For example, you could fade out and then fade back in portions of the interface.

Implementing this kind of advanced user interaction is entirely your responsibility if you are using the script services approach. If, however, you use the partial rendering approach and install the AJAX Control Toolkit, you can rely on an extremely cool extender control: the UpdatePanelAnimation extender.

An extender control is an ASP.NET server control that provides additional behavior to a variety of existing server controls. When you need a special behavior from a control, you code the behavior in an extender and then bind together the extender and the original control in the same page. Suppose you want to filter out non-digit characters in a textbox. You can develop an ad hoc TextBox control or you can just use the plain old TextBox control extended with the proper behavior. The advantage is that the same behavior can be reused for multiple types of controls and, further, that multiple behaviors can be freely combined to enrich a particular instance of a server control.

The UpdatePanelAnimation extender was designed to add a bit of animation to any updatable regions in an ASP.NET AJAX partial rendering page. You declaratively associate the animation you want the page to use while the postback proceeds. The extender works in conjunction with the script-based animation framework available through the AJAX Control Toolkit. Figure 6 shows sample code you can use to animate an updatable region.

Figure 6 Using the UpdatePanelAnimation Extender (Close Figure 6)
<asp:UpdatePanel ID="UpdatePanel1" runat="server"
    UpdateMode="Conditional">
     <ContentTemplate>
         <asp:Button ID="Button1" runat="server" Text="Load" ... />
         <div id="Panel1">
            ...
         </div>
     </ContentTemplate>
     <Triggers>
         ...
     </Triggers>
</asp:UpdatePanel>

<act:UpdatePanelAnimationExtender ID="UpdatePanelAnimation1" runat="server"
    TargetControlID="UpdatePanel1">
    <Animations>
        <OnUpdating>
            <Sequence>
                <EnableAction AnimationTarget="Button1" Enabled="false" />
                <FadeOut AnimationTarget="Panel1" minimumOpacity=".3" />
            </Sequence>
        </OnUpdating>
        <OnUpdated>
            <Sequence>
                <FadeIn AnimationTarget="Panel1" minimumOpacity=".3" />
                <EnableAction AnimationTarget="Button1" Enabled="true" />
            </Sequence>
        </OnUpdated>
    </Animations>
</act:UpdatePanelAnimationExtender>

You first bind the extender to a particular UpdatePanel control in the page and then configure the animation. The code in Figure 6 sets up a sequence of two actions before the postback occurs (the OnUpdating tag) and another sequence of two actions when the postback has completed (the OnUpdated tag). EnableAction, FadeOut, and FadeIn are all predefined animations supported by the AJAX Control Toolkit. FadeOut and FadeIn use CSS styles to fade out and in the DOM subtree they apply to—Panel1 in this example. The EnableAction animation simply disables or enables the specified target DOM element.

Compared to direct scripting, the UpdatePanel animation extender supports declarative programming and allows you to do more than just graying out elements. The extender, though, is limited to pages that use partial rendering. For more information, take a look at the AJAX Control Toolkit home page at codeplex.com/AtlasControlToolkit.

Disabling the Whole Page

As I mentioned, the portion of the UI to be disabled depends on the control the user clicked and, more generally, the required action. In a partial rendering context, though, only one request can be executed at a time. Identifying and disabling only the portion of the interface affected by the pending operation would be too much work and, more importantly, it wouldn't guard against undesired clicking. I'll discuss this in more detail in a moment. For now, suffice it to say that disabling the whole page without the burden of figuring out which controls should be grayed out is a simpler and more effective solution.

The idea is to create a DIV tag as large as the client area of the browser and use it to cover everything in the page. In this way, any clicking will be captured by this DIV and won't reach the underlying HTML elements. You can even associate the DIV with some styles to gray out the page while the operation is processing. Figure 7 shows the JavaScript code used to accomplish this task.

Figure 7 Covering the UI with a DIV (Close Figure 7)
<script type="text/javascript">
    var _backgroundElement = document.createElement("div");

    function pageLoad()
    {
        var manager = Sys.WebForms.PageRequestManager.getInstance();
        manager.add_beginRequest(OnBeginRequest);
        manager.add_endRequest(OnEndRequest);

    // pageContent is the parent of the new DIV
        $get("pageContent").appendChild(_backgroundElement);
    }

    function OnBeginRequest(sender, args)
    {
        EnableUI(false);
    }

    function OnEndRequest(sender, args)
    {
        EnableUI(true);
    }

    function EnableUI(state)
    {
        if (!state)
        {
            _backgroundElement.style.display = '';
            _backgroundElement.style.position = 'fixed';
            _backgroundElement.style.left = '0px';
            _backgroundElement.style.top = '0px';

            var clientBounds = this._getClientBounds();
            var clientWidth = clientBounds.width;
            var clientHeight = clientBounds.height;
            _backgroundElement.style.width =
               Math.max(Math.max(document.documentElement.scrollWidth,
               document.body.scrollWidth), clientWidth)+'px';
            _backgroundElement.style.height =
               Math.max(Math.max(document.documentElement.scrollHeight,
               document.body.scrollHeight), clientHeight)+'px';
            _backgroundElement.style.zIndex = 10000;
            _backgroundElement.className = "modalBackground";
        }
        else
        {
            _backgroundElement.style.display = 'none';
        }
    }

    function _getClientBounds()
    {
        var clientWidth;
        var clientHeight;
        switch(Sys.Browser.agent) {
            case Sys.Browser.InternetExplorer:
                clientWidth = document.documentElement.clientWidth;
                clientHeight = document.documentElement.clientHeight;
                break;
            case Sys.Browser.Safari:
                clientWidth = window.innerWidth;
                clientHeight = window.innerHeight;
                break;
            case Sys.Browser.Opera:
                clientWidth = Math.min(window.innerWidth,
                              document.body.clientWidth);
                clientHeight = Math.min(window.innerHeight,
                               document.body.clientHeight);
                break;
            default:  // Sys.Browser.Firefox, etc.
                clientWidth = Math.min(window.innerWidth,
                              document.documentElement.clientWidth);
                clientHeight = Math.min(window.innerHeight,
                               document.documentElement.clientHeight);
                break;
        }
        return new Sys.UI.Bounds(0, 0, clientWidth, clientHeight);
    }
</script>

A global JavaScript variable is declared to represent the DIV tag. Once initialized, the DIV is added to the DOM tree at the specified position. In this case, it is defined as the child of the DIV tag that wraps the whole visible interface. The BeginRequest handler configures and displays the DIV, thus disabling the visible interface. The EndRequest handler just hides the DIV, thus restoring the UI:

function OnBeginRequest(sender, args)
{
    EnableUI(false);
}

function OnEndRequest(sender, args)
{
    EnableUI(true);
}

The DIV tag is fixed at 0,0 position and its width and height are dynamically calculated in a browser-specific manner. The size of the DIV reflects the current size of the browser's client area. The getClientBounds function shows an example of script programming that works around the differences in the object model of various browsers.

As you can see, the code uses some classes built in the Microsoft® AJAX client library. I borrowed the code snippet from the source code of the ModalPopup extender—one of the coolest extenders in the AJAX Control Toolkit. In particular, the ModalPopup extender allows a page to display content to the user with modality—that is, preventing the user from interacting with the rest of the page. The net effect is the same as window.alert, but the interface—the hierarchy of controls being displayed—is entirely up to you.

Figure 8 shows a page that uses a modal DIV to disable the entire UI during a partial rendering operation.

Figure 8 Disabling the Entire Page during Partial Rendering
Figure 8 Disabling the Entire Page during Partial Rendering(Click the image for a smaller view)
Figure 8 Disabling the Entire Page during Partial Rendering
Figure 8 Disabling the Entire Page during Partial Rendering (Click the image for a larger view)

One AJAX Postback at a Time

Since partial rendering lets you add AJAX functionality to classic ASP.NET 2.0 pages without modifying the programming model, the server lifecycle of the page is fully preserved. This holds true for postback events, loading, rendering, and view state. The amount of traffic for a typical partial rendering call is smaller than that for a classic ASP.NET call, but the size of the view state is a constant and can only be reduced through page-specific programming techniques.

With partial rendering, an ASP.NET page gains the continuous feel of an AJAX page, but maintains the traditional user interface. Seen from the client-side, a partial rendering page behaves as if it is part of a single, sequential UI. Only one operation can be pending at a time, even when two concurrent operations are possible from a business viewpoint.

Imagine you have a page that displays any sort of information about a customer. This page contains two buttons—one for downloading order information and one for displaying customer details. From a pure business perspective, there's nothing preventing the user from clicking the two buttons in sequence to generate two successive requests, with the second request starting before the first has completed. The two requests need the same input—the customer ID—and both are read-only, meaning there is no danger of interfering with the status of the back-end database. Nevertheless, these two calls can't run concurrently in a partial-rendering page because partial rendering is based on the classic ASP.NET programming model—each request returns markup that includes an updated view state for the whole page. If you let two requests run concurrently, you put the consistency of the page's view state at risk. Two requests sent in quick sequence send the same view state to the server, but each request returns a different view state, updated for the controls involved with the processing of the request. This means the last request to return overwrites the changes made when the first request returned. Thus, partial-rendering AJAX pages still work according to the stop-and-go pattern.

To avoid these types of problems, the page request manager object automatically kills any request that is pending when a new request is placed. In light of this behavior, it is extremely important that you disable the entire UI in partial rendering pages. This is to ensure that users can't interact with the whole page for the duration of the current request. As you can see, this statement directly contradicts one of the key benefits of AJAX—asynchronous interaction.

High-Priority Calls

Certain questions come up when examining partial rendering, such as if you can't run two requests concurrently due to the need to preserve the consistency of the view state, then why hasn't the team implemented an algorithm to merge two partial view states on the client? This seems like a reasonable question. However, it is not an option due at least in part to the view state's internal structure, which includes encoded content, a hash value, and a server-only security value.

Combining two view states on the client is technically possible, but it would require significant changes to the current structure of the view state and the ASP.NET runtime. Moreover, it would raise security issues by making script code available to virtually everybody, allowing them to prepare ad hoc view state content for replay and cross-site scripting attacks.

Another common question has to do with the last-win policy that the page request manager implements when it kills the pending request whenever a new request is received. I find that developers tend to feel constrained when someone else makes a decision about a programming point. So it seems that many developers—myself, included—don't like the imposed last-win policy for AJAX postbacks. What about a first-win policy where basically the current request is given precedence and successive requests are canceled or queued until the first request finishes?

The ASP.NET AJAX framework provides the tools needed to implement this pattern, as you'll see in a moment. But are you sure you really need this option? The fact is that if you use partial rendering, you should be ready to deliver your pages one request at a time. The easiest and safest way to accomplish this is to disable the whole UI to prevent users from clicking.

If, for some reason, you don't think you can disable the entire user interface, then knowing how to implement a first-win approach might deliver the flexibility you need. The idea is to write some script code that intercepts the beginning of each partial rendering request—the initializeRequest event of the page request manager. In the handler, you check run-time conditions to determine whether you really want to cancel the request and then instruct the manager accordingly:

function OnInitializeRequest(sender, args)
{
    var manager = sender;
    if (manager.get_isInAsyncPostBack() &&
        args.get_postBackElement().id === "Button1")
    {
        args.set_cancel(true);
    }
}

The sample code checks whether another request is being served and whether this is a request with a lower priority. If so, you can cancel the request by setting the cancel property to true on the event data structure. According to the previous code, the latest request is gone. However, you might write code to implement an internal queue for pending requests. For more information on high-priority calls, check out ajax.asp.net/docs/tutorials/ExclusiveAsyncPostback.aspx.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Introducing ASP.NET AJAX (Microsoft Press, 2007). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can reach him at cutting@microsoft.com or join his blog at weblogs.asp.net/despos.


Team System: Custom check-in policies

Team System
Custom check-in policies
Brian A. Randell

Code download available at: TeamSystem2007_11.exe (166KB)
Contents
Check-In Notes and Policies
Creating Your Own Policy
Debugging Policy Assemblies
Enhancing the Custom Policy
Adding Edit Support

I
n the past three editions of this column, I've explored the Team Foundation Server (TFS) version control and work item tracking APIs. I've used the APIs to build a Microsoft® Word 2003 add-in that provides support for check-ins and work item association of Word documents similar to what's available via the Team Explorer in Visual Studio® 2005. In this column, I'll dig into check-in notes and policies. You'll learn how check-in notes work and how to write your own custom policy implementations. In a later column, I'll add this support to the Word add-in.

Check-In Notes and Policies

Check-in notes are free-form text fields that allow you to add categorized string data to a check-in. Check-in notes are defined at the team project level and can be made mandatory as part of a check-in operation. The default project templates provided by Microsoft define three check-in notes: Code Reviewer, Security Reviewer, and Performance Reviewer. None of them is mandatory; you're free to define your own check-in notes by either using the Team Explorer and/or APIs after you create a team project or by modifying the VersionControl.xml file in your process template before you create a team project. You define a check-in note by defining a label of up to 64 characters, specifying whether or not the note is required at check-in, and specifying the note's display position on the check-in form. At check-in time, the value of a note can be up to 2048 + (230 – 1) characters. The first 2048 characters are stored in an nvarchar column in the database—anything over that is stored in an ntext column.

Check-in policies enforce rules defined by your team project administrator. A policy can range in complexity from something as simple as ensuring you enter a comment at check-in, to something more complicated such as performing static code analysis and running tests. In the 2005 release of TFS, three policies were included out of the box. Later, as part of the Microsoft Visual Studio 2005 Team Foundation Server Power Tool, Microsoft added four additional check-in policies. Figure 1 provides the details about all seven. (And yes, that's "Tool," not "Tools." See msdn2.microsoft.com/aa718351.)

Figure 1 Policies Provided by Microsoft (Close Figure 1)

Policy | Release | Policy Requirements
Work Items | RTM | Associate at least one work item in the check-in. There are no configuration options.
Code Analysis | RTM | Run static code analysis (which implies a clean compile) before check-in. You control whether the policy applies to managed code and/or C++ code.
Testing | RTM | Run the tests specified by the policies' test list before check-in. You control the list of tests that policy runs.
Comment Association | Power Tool | Enter a comment at check-in. There are no configuration options.
Forbidden Patterns | Power Tool | Allows you to define a set of files you do not want checked in to the repository. You configure the policy by specifying a simple file extension or use a regular expression.
Work Item Query | Power Tool | Associates a work item from the specified query with your check-in. You configure the policy by selecting a team query from your team project.
Custom Path | Power Tool | Works in tandem with another policy. It lets you selectively apply the "buddy" policy to a subset of your team project's source code control tree.

You implement a policy by creating a managed class library that exposes a serializable public class that implements two well-known interfaces: IPolicyDefinition and IPolicyEvaluation. You install the policy assembly by first copying it to a workstation with the Team Explorer bits installed. Next, you register the assembly by adding an entry in the Windows® registry at HKEY_LOCAL_ MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\SourceControl\Checkin Policies. You create a string value where the Name is the file name of your policy assembly, without the .dll extension, and the Data is the fully qualified path to the assembly on the local workstation (see Figure 2).

Figure 2 Registry Keys
Figure 2 Registry Keys(Click the image for a smaller view)
Figure 2 Registry Keys
Figure 2 Registry Keys (Click the image for a larger view)

Once the policy is installed, you will enable it for each team project, typically by using the Team Explorer's Source Control Settings dialog. You access this dialog by first selecting a valid team project in the Team Explorer tool window and then navigating to Team | Team Project Settings | Source Control and choosing the Check-in Policy tab (see Figure 3).

Figure 3 Check-In Policy Tab
Figure 3 Check-In Policy Tab(Click the image for a smaller view)
Figure 3 Check-In Policy Tab
Figure 3 Check-In Policy Tab (Click the image for a larger view)

The first time you choose the Add button from the Source Control Settings dialog, Team Explorer enumerates the list of registered assemblies on the local workstation. (Once it has done this, it actually enumerates the list each time you open the Source Control Settings dialog). If you click the Add button, Team Explorer presents the list of installed policy assemblies (see Figure 4) as well as a description of the policy selected.

Figure 4 Add Check-In Policy Dialog
Figure 4 Add Check-In Policy Dialog

Once you select a check-in policy from the Add Check-in Policy dialog, Team Explorer will query the CanEdit property of the policy's IPolicyDefinition implementation. If CanEdit returns true, then Team Explorer calls the policy's Edit method, allowing the policy to display a dialog for additional configuration. After you've configured the policy and closed the Source Control Settings dialog, Team Explorer calls the SetCheckinPolicies method from an instance of the TeamProject object, which serializes a bit of metadata about the policy assembly as well as an instance of the policy class to the Team Foundation Server database.

Note that TFS knows nothing of your policy assembly until you enable it for a particular team project. Once you've enabled a policy, though, you need to install the policy assembly on the workstation of every developer who will be performing a check-in. TFS currently does not provide any automatic deployment mechanism.

Creating Your Own Policy

To create your own policy, you need to have the Microsoft .NET Framework 2.0 and Team Explorer installed. While you can create an assembly without an installation of Visual Studio 2005, I'm going to assume you're using at least Visual Studio 2005 Professional edition. First, you create a new Visual Basic® Class Library project. For this example, you're going to create a policy assembly that scans .vb or .cs files for "bad" words. I've called the assembly MSDNMag TeamSystem.CheckinPolicies. Once you've created the project, rename Class1.vb to BadWords.vb and mark the class Serializable.

You'll need to add a reference to the Microsoft.TeamFoundation.VersionControl.Client.dll assembly. This assembly is installed as a part of the Visual Studio 2005 SDK, and contains the definitions for IPolicyDefinition and IPolicyEvaluation. In addition, you'll find an abstract base class, PolicyBase, that you can use to get started a bit more quickly. In this example, however, you'll explicitly implement the interfaces one at a time in your own class.

In the BadWords.vb class file, add an imports statement for Microsoft.TeamFoundation.VersionControl.Client. Then implement the IPolicyDefinition interface. Visual Studio 2005 will stub out the six members of this interface. Figure 5 provides the implementation for this interface.

Figure 5 IPolicyDefinition for the BadWords Policy (Close Figure 5)
Imports Microsoft.TeamFoundation.VersionControl.Client

<Serializable()> _
Public Class BadWords
  Implements IPolicyDefinition

  Public ReadOnly Property CanEdit() As Boolean _
    Implements IPolicyDefinition.CanEdit
    Get
      Return False
    End Get
  End Property

  Public ReadOnly Property Description() As String _
    Implements IPolicyDefinition.Description
    Get
      Return "Checks code for bad words."
    End Get
  End Property

  Public Function Edit(ByVal policyEditArgs As IPolicyEditArgs) _
    As Boolean Implements IPolicyDefinition.Edit

    Throw New NotImplementedException("Edit is not implemented")
  End Function

  Public ReadOnly Property InstallationInstructions() As String _
    Implements IPolicyDefinition.InstallationInstructions
    Get
      Using sw As New IO.StringWriter()
        With sw
          .Write("You need to install ")
          .Write("BrianRandell.MSDNMag.CheckinPolicies.dll ")
          .Write("on your local workstation." & .NewLine())
          .WriteLine("Visit the Team Project for more info.")
        End With

        Return sw.ToString()
      End Using
    End Get
  End Property

  Public ReadOnly Property Type() As String _
      Implements IPolicyDefinition.Type
    Get
      Return "Brian Randell-MSDNMag Check for Bad Words""
    End Get
  End Property

  Public ReadOnly Property TypeDescription() As String _
    Implements IPolicyDefinition.TypeDescription
    Get
      Return "This policy checks .vb and .cs files for bad words."
    End Get
  End Property
End Class

Most of the code in Figure 5 is self-explanatory. The interface provides string values that describe the policy and provides the necessary hooks, CanEdit and Edit, should you choose to have a policy that supports additional configuration. Note that the Type property represents the display string that is presented in the Add Check-in Policy dialog. To get started, you need to hard-code the list of bad words. Later, you can implement the CanEdit and Edit methods.

After you've implemented IPolicyDefinition, you need to implement IPolicyEvaluation. This interface defines four methods (Activate, DisplayHelp, Evaluate, and Initialize), and a single event (PolicyStateChanged). The policy evaluation framework calls Initialize after your policy class instance is created. This method accepts an IPendingCheckin reference, which provides that policy assembly with access to the data, such as pending changes and work items, related to a pending check-in. Evaluate is the heart of the policy assembly. The check-in framework executes this method when the user clicks the Policies button or attempts a check-in or shelving operation. If Evaluate returns a non-empty list of PolicyFailure instances, the framework cancels the check-in and presents the list of errors. If you double-click on an error message, the framework calls Activate, allowing the policy class to display details about the error. Finally, if you press F1 with an error selected, the framework executes the DisplayHelp method, which can display a help file, navigate to a Web page, or present a custom UI. The policy class uses the PolicyStateChanged event to notify the framework that changes have occurred and that it should call Evaluate again. Figure 6 provides the initial implementation for IPolicyEvaluation.

Figure 6 IPolicyEvaluate for the BadWords Policy (Close Figure 6)
Imports Microsoft.TeamFoundation.VersionControl.Client
Imports System.IO

<Serializable()> _
Public Class BadWords
  Implements IPolicyDefinition, IPolicyEvaluation

' IPolicyDefinition elided (see Figure 5)

  Private badWords() As String = {"Dag", "Darn", "Doh"}
  <NonSerialized()> _
  Private m_pendingCheckin As IPendingCheckin = Nothing

  Public Sub Activate(ByVal failure As PolicyFailure) _
    Implements IPolicyEvaluation.Activate
    ' Do Nothing for Now
  End Sub

  Public Sub DisplayHelp(ByVal failure As PolicyFailure) _
    Implements IPolicyEvaluation.DisplayHelp
    ' Do Nothing for Now
  End Sub

  Public Function Evaluate() As PolicyFailure() _
    Implements IPolicyEvaluation.Evaluate

    Dim fileContents As String = Nothing
    Dim failureMessage As String = Nothing
    Dim failures As New List(Of PolicyFailure)

    For Each pc As PendingChange In _
      m_pendingCheckin.PendingChanges.CheckedPendingChanges

      ' Only check .vb and .cs files
      If pc.FileName.EndsWith(".vb") OrElse _
        pc.FileName.EndsWith(".cs") Then

        Using sr As New StreamReader(pc.FileName)
          fileContents = sr.ReadToEnd()
        End Using
        For Each s As String In badWords
          If fileContents.Contains(s) Then
            failureMessage = pc.FileName & " has a bad word."
            Dim pf As New PolicyFailure(failureMessage, Me)
            failures.Add(pf)
            Exit For
          End If
        Next
      End If
    Next

    Return failures.ToArray()
  End Function

  Public Sub Initialize(ByVal pendingCheckin As IPendingCheckin) _
    Implements IPolicyEvaluation.Initialize

    If (Not Me.m_pendingCheckin Is Nothing) Then
      Throw New InvalidOperationException("Already Active")
    End If
    If disposedValue Then
      Throw New ObjectDisposedException(Nothing)
    End If

    Me.m_pendingCheckin = pendingCheckin
  End Sub

  Public Event PolicyStateChanged(ByVal sender As Object, _
    ByVal e As PolicyStateChangedEventArgs) _
    Implements IPolicyEvaluation.PolicyStateChanged

  ' IDisposable implementation elided

End Class

The initial implementation is very straightforward—the policy instance stores a reference to the IPendingCheckin reference provided via the Initialize method. In the Evaluate method, the code enumerates the CheckedPendingChanges collection, retrieving a PendingChange instance for each checked item. If the item is a file that ends with either a .vb or .cs extension, the code loads the file contents into a string variable using a StreamReader. Then the code scans the file contents using the String.Contains method for each bad word, one at a time. If the code finds a bad word, it creates a new PolicyFailure instance, adds it to the failures List(Of T) collection, and continues to the next pending change. Once it has scanned all the files, it returns the failures collection to the policy framework that invoked Evaluate.

After you have added the necessary code to support IPolicyEvaluation, you need to compile the assembly. You have to add the necessary data to the registry so Visual Studio will load the assembly when you click the Add button in the Source Code Control Settings dialog. Whenever you modify the list of installed policies, you'll need to restart any running instances of Visual Studio so that it will see the changes. Once you have everything configured, turn on the policy and then modify a file under source control, entering a bad word. Then, open the Pending Changes dialog and click the Policy Warnings button or attempt a check-in. You should see something similar to Figure 7.

Figure 7 Policy Violation Warning
Figure 7 Policy Violation Warning(Click the image for a smaller view)
Figure 7 Policy Violation Warning
Figure 7 Policy Violation Warning (Click the image for a larger view)

Debugging Policy Assemblies

Debugging the custom policy is actually quite easy. First, if you've enabled your custom policy, remove it from your Team Project list. You'll make your life much easier if you only work with your policy assembly in a private Team Project. Naturally, at some point you'll want to test your policy assembly with any other policies you choose to enable, but reducing the problem surface area when you're debugging just makes sense. Once you've removed all policies, shut down all running instances of Visual Studio 2005 and leave the data you added earlier in the registry. Then, restart one instance of Visual Studio 2005. Load your custom policy project and access the Project designer (right-click on your Project in the Solution Explorer and select Properties). Activate the Debug tab and change the Start Action to Start External Program. You'll need to provide the fully qualified path to Visual Studio 2005, typically C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv.exe. Save your changes and close the designer.

Once you've done this, set breakpoints on IPolicyDefinition.Type and IPolicyEvaluate.Evaluate. Start debugging by pressing F5 or by choosing Debug | Start Debugging from the menu. A second instance of Visual Studio will start and, in this instance, open the project you're using to test for bad words. Then access the Source Control Settings dialog and add your custom policy. When you click the Add button, the first instance of Visual Studio will become the active window as Visual Studio accesses the Type property of your policy. When you close the settings dialog, the policy framework will execute your policy's Evaluate method, which should, in turn, reactivate the first instance of Visual Studio and allow you to start debugging. At the end of the debugging session, return to the Source Control Settings dialog and remove your policy. You will have a chance to debug your IDisposable implementation here.

When you're finished, close the second instance of Visual Studio. At this point, you can modify your policy assembly and repeat the process. You should only enable the policy in the second instance of Visual Studio. Otherwise, you'll need to unload all instances to recompile.

Enhancing the Custom Policy

While the current implementation of the custom policy works, it's not very feature-rich. It doesn't provide any details about where the bad words are located. In addition, you've hardcoded the list of bad words. To rectify this, you'll first need to supply a richer implementation for IPolicyEvaluate.Evaluate. Then you can provide an actual implementation for the Activate method. Finally, you'll need to supply an implementation for IPolicyDefinition.Edit. Figure 8 provides the new code for Evaluate.

Figure 8 Updated Implementation of Evaluate (Close Figure 8)
Imports Microsoft.TeamFoundation.VersionControl.Client
Imports System.IO

<Serializable()> _
Public Class BadWords
  Implements IPolicyDefinition, IPolicyEvaluation

  Private badWords() As String = {"Dag", "Darn", "Doh"}
  <NonSerialized()> _
  Private m_pendingCheckin As IPendingCheckin = Nothing
  <NonSerialized()> _
  Private badWordFound As Boolean = False

  ' IPolicyDefinition elided
  ' IPolicyEvaluation Activate and DisplayHelp elided

  Public Function Evaluate() As PolicyFailure() _
    Implements IPolicyEvaluation.Evaluate

    Dim fileContents As String = Nothing
    Dim failureMessage As String = Nothing
    Dim wordData As List(Of WordInfo)
    Dim failures As New List(Of PolicyFailure)

    For Each pc As PendingChange In _
      m_pendingCheckin.PendingChanges.CheckedPendingChanges

      ' Only check .vb and .cs files
      If pc.FileName.EndsWith(".vb") OrElse _
        pc.FileName.EndsWith(".cs") Then

        badWordFound = False
        wordData = GetWordInfo(pc.LocalItem)
        If badWordFound Then
          failureMessage = _
            String.Format("{0} has at least one bad word." & _
            " Double-click for details", pc.FileName)

          Dim bpf As New BadWordPolicyFailure(failureMessage, _
            Me, New CheckedFileInfo(pc.FileName, wordData, pc))

          failures.Add(bpf)
        End If
      End If
    Next

    Return failures.ToArray()
  End Function

  ' IPolicyEvaluation Initialize elided
  ' IDisposable implenation elided

  Friend Function GetWordInfo(ByVal fileName As String) _
    As List(Of WordInfo)

    Dim lines As String() = File.ReadAllLines(fileName)
    Dim wordData As New List(Of WordInfo)

    For Each word As String In badWords
      Dim item As WordInfo = New WordInfo(word)
      wordData.Add(item)

      For i As Integer = 0 To lines.Length - 1
        If lines(i).Contains(word) Then
          item.LineList.Add(i + 1)
          item.LineData.Add(lines(i))
        End If
      Next i
      If item.LineList.Count > 0 Then
        badWordFound = True
      End If
    Next

    Return wordData
  End Function
End Class

The new implementation of Evaluate uses a set of new helper classes, shown in Figure 9. WordInfo stores both the line number and actual line of code containing the bad word. The code creates an instance of WordInfo for each bad word. CheckedFileInfo contains the name of the file checked, a collection of WordInfo objects, and a reference to the PendingChange item. Finally, BadWordPolicyFailure, which derives from PolicyFailure, holds a reference to an instance of CheckedFileInfo. This class exists because the default PolicyFailure class does not provide an easy way to associate a policy failure with supporting metadata that the Activate method needs to provide a rich user experience.

Figure 9 Helper Classes for Evaluate (Close Figure 9)
Public Class WordInfo
  Public Word As String
  Public LineList As New List(Of Integer)
  Public LineData As New List(Of String)

  Public Sub New(ByVal word As String)
    Me.Word = word
  End Sub
  Public Overrides Function ToString() As String
    Return Word
  End Function
End Class

Public Class CheckedFileInfo
  Public FileName As String
  Public WordData As List(Of WordInfo)
  Public PendingItem As PendingChange

  Public Sub New(ByVal FileName As String, _
    ByVal WordData As List(Of WordInfo), _
    ByVal PendingItem As PendingChange)

    Me.FileName = FileName
    Me.WordData = WordData
    Me.PendingItem = PendingItem
  End Sub
End Class

Public Class BadWordPolicyFailure
  Inherits PolicyFailure

  Public FailureData As CheckedFileInfo

  Public Sub New(ByVal Message As String, _
    ByVal Policy As IPolicyEvaluation, _
    ByVal FailureData As CheckedFileInfo)

    MyBase.New(Message, Policy)
    Me.FailureData = FailureData
  End Sub
End Class

With these new classes, the new Evaluate method works slightly differently. The code still enumerates the CheckedPendingChanges collection, looking for only .vb and .cs files. However, the GetWordInfo helper method does the work of actually loading the source file and looking for bad words. This method builds a collection containing a WordInfo for each bad word on this list. If GetWordInfo finds at least one bad word, the Evaluate method creates an instance of CheckedFileInfo and passes it to a new instance of BadWordPolicyFailure, which is added to the failures collection. Then it continues checking each file.

Next, you need to provide an implementation for the Activate method. When you double-click on a policy violation message, you want the policy to display a dialog listing the bad words, what lines they occur on, and the actual line of code with the bad word. The sample code download provides a simple form, frmFileDetails, to get you started.

In Figure 10 you can see the new code for the Activate method. The first thing the code does is TryCast the provided PolicyFailure reference to the derived BadWordPolicyFailure type. If this succeeds, the code displays frmFileDetails as a dialog (see Figure 11 for an example).

Figure 10 Activate Method Implementation (Close Figure 10)
Public Sub Activate(ByVal failure As PolicyFailure) _
  Implements IPolicyEvaluation.Activate

  Dim bpf As BadWordPolicyFailure = _
    TryCast(failure, BadWordPolicyFailure)

  If bpf IsNot Nothing Then
    Using frm As New frmFileDetails(bpf)
      frm.ShowDialog()
    End Using
  Else
    MsgBox("The Policy Failure object provided is invalid.", _
      MsgBoxStyle.Exclamation, "Warning")
  End If
End Sub
Figure 11 Policy Failure Message
Figure 11 Policy Failure Message(Click the image for a smaller view)
Figure 11 Policy Failure Message
Figure 11 Policy Failure Message (Click the image for a larger view)

Adding Edit Support

The last feature you need to add is a dialog that allows the Project Administrator to define the list of bad words. There are three tasks for you to complete here. First, you need to provide a storage mechanism for the list of bad words. Next, you must provide a UI to edit the list of bad words. Finally, you need to modify the policy to load the bad words from the storage location instead of using the hardcoded array. None of these tasks is specific to policy assemblies—they're all just standard code. I've provided an implementation in the code sample. The sample stores the list of bad words in a text file located in the same directory as the add-in. At edit time, the Project Administrator must have read/write access to the bad-words file, though consumers of the policy will only need read access.

At this point, you should feel comfortable with check-in notes and policies. The framework Microsoft has provided for check-in policies is rich and flexible. Going forward, I look for Microsoft to improve the installation experience. In the next edition of this column, I'll dig into how to programmatically work with check-in notes and policies from custom code and the Word add-in.

Send your questions and comments to mmvsts@microsoft.com.

Brian A. Randell is a Senior Consultant with MCW Technologies LLC. Brian spends his time speaking, teaching, and writing about Microsoft technologies. He is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. You can contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


Foundations: Synchronization Contexts in WCF

Foundations
Synchronization Contexts in WCF
Juval Lowy

Code download available at: Foundations2007_11.exe (241KB)
Contents
What Are .NET Synchronization Contexts?
WCF and Synchronization Contexts
Service Custom Synchronization Context
Thread Affinity Synchronization Context
Priority Processing
Callbacks and Synchronization Context
Why Use Synchronization Contexts?

O
ne of the more useful features of the Windows® Communication Foundation (WCF) is its reliance on the Microsoft® .NET Framework synchronization context to marshal calls to the service instance (or to a callback object). This mechanism provides for both productivity-oriented development and for powerful extensibility. In this column, I describe briefly what synchronization contexts are and how WCF uses them, and then proceed to demonstrate various options for extending WCF to use custom synchronization contexts, both programmatically and declaratively. In addition to seeing the merits of custom synchronization contexts, you will also see some advanced .NET programming as well as WCF extensibility techniques.

What Are .NET Synchronization Contexts?

The .NET Framework 2.0 introduced a little-known feature called the synchronization context, defined by the class SynchronizationContext in the System.Threading namespace:

public delegate void SendOrPostCallback(object state);

public class SynchronizationContext
{
   public virtual void Post(SendOrPostCallback callback,object state);
   public virtual void Send(SendOrPostCallback callback,object state);
   public static void SetSynchronizationContext
(SynchronizationContext context);
   public static SynchronizationContext Current
   {get;}
   //More members
}

The synchronization context is stored in the thread local storage (TLS). Every thread created with the .NET Framework 2.0 may have a synchronization context (similar to an ambient transaction) obtained via the static Current property of SynchronizationContext. Current may return null if the current thread has no synchronization context. The synchronization context is used to bounce a method call between a calling thread and a target thread or threads, in case the method cannot execute on the original calling thread. The calling thread wraps the method it wants to marshal to the other thread (or threads) with a delegate of the type SendOrPostCallback, and provides it to the Send or Post methods, for synchronous or asynchronous execution respectively. You can associate a synchronization context with your current thread by calling the static method SetSynchronizationContext.

By far, the most common use of a synchronization context is with UI updates. With all multithreaded Windows technologies, from MFC to Windows Forms to WPF, only the thread that created a window is allowed to update it by processing its messages. This constraint has to do with the underlying use of the Windows message loop and the thread messaging architecture. The messaging architecture creates a problem when developing a multithreaded UI application—you would like to avoid blocking the UI when executing lengthy operations or receiving callbacks. This, of course, necessitates the use of worker threads, yet those threads cannot update the UI directly because they are not the UI thread.

In order to address this problem in the .NET Framework 2.0, the constructor of any Windows Forms-based control or form checks to see if the thread it is running on has a synchronization context, and if it does not have one, the constructor attaches a new synchronization context (called WindowsFormsSynchronizationContext). This dedicated synchronization context converts all calls to its Post or Send methods into Windows messages and posts them to the UI thread message queue to be processed on the correct thread. The Windows Forms synchronization context is the underlying technology behind the commonly used BackgroundWorker helper control.

WCF and Synchronization Contexts

By default, all WCF service calls (and callbacks) execute on threads from the I/O Completion thread pool. That pool has 1,000 threads by default, none of them under the control of your application. Now imagine a service that needs to update some user interface. The service should not access the form, control or window directly because it is on the wrong thread. To deal with that, the ServiceBehaviorAttribute offers the UseSynchronizationContext property, defined as:

[AttributeUsage(AttributeTargets.Class)]
public sealed class ServiceBehaviorAttribute : ...
{
   public bool UseSynchronizationContext
   {get;set;}
   //More members
}

The default value of UseSynchronizationContext is set to true. To determine which synchronization context the service should use, WCF looks at the thread that opened the host. If that thread has a synchronization context and UseSynchronizationContext is set to true, then WCF automatically marshals all calls to the service to that synchronization context. There is no explicit interaction with the synchronization context required of the developer. WCF does that by providing the dispatcher of each endpoint with a reference to the synchronization context, and the dispatcher uses that synchronization context to dispatch all calls.

For example, suppose the service MyService (defined in Figure 1) needs to update some UI on the form MyForm. Because the host is opened after the form is constructed, the opening thread already has a synchronization context, and so all calls to the service will automatically be marshaled to the correct UI thread.

Figure 1 Using the UI Synchronization Context (Close Figure 1)
[ServiceContract]
interface IMyContract
{...}

class MyService : IMyContract
{
   /* some code to update MyForm */
}

class MyForm : Form
{...}

static class Program
{
   static void Main()
   {
      Form form = new MyForm();//Sync context established here

      ServiceHost host = new ServiceHost(typeof(MyService));
      host.Open();

      Application.Run(form);

      host.Close();
   }
}

If you were to simply open the host in Main and use the code generated by the Windows Forms designer, you would not benefit from the synchronization context, since the host is opened without it present:

//No automatic use of synchronization context
static void Main()
{
   ServiceHost host = new ServiceHost(typeof(MyService));
   host.Open();

   Application.Run(new MyForm());//Synchronization context
                                 //established here

   host.Close();
}

Service Custom Synchronization Context

Utilizing a custom synchronization context by your service or resources has two aspects to it: implementing the synchronization context and then installing it. The source code accompanying this column contains my ThreadPoolSynchronizer class, which is defined as:

public class ThreadPoolSynchronizer : SynchronizationContext,IDisposable
{
   public ThreadPoolSynchronizer(uint poolSize);
   public ThreadPoolSynchronizer(uint poolSize,string poolName);

   public void Dispose();
   public void Close();
   public void Abort();
}

ThreadPoolSynchronizer marshals all calls to a custom thread pool, where the calls are first queued up, then multiplexed on the available threads. The size of the pool is provided as a construction parameter. If the pool is maxed out, the call will be pending in the queue until a thread is available. You can also provide a pool name (which will be the prefix of the name of the threads in the pool). Disposing or closing ThreadPoolSynchronizer kills all threads in the pool gracefully, that is, after allowing the engaged threads to complete their task. The Abort method causes a less-than-graceful shutdown, as it terminates all threads abruptly. Implementing ThreadPoolSynchronizer had nothing to do with WCF, so I chose not to include the code in this column. The classic use for a custom thread pool is with a server application that needs to maximize its throughput by controlling the underlying worker threads and their assignments.

To associate your service with the custom thread pool, you can manually attach ThreadPoolSynchronizer to the thread opening the host:

SynchronizationContext syncContext = new ThreadPoolSynchronizer(3);

SynchronizationContext.SetSynchronizationContext(syncContext);

using(syncContext as IDisposable)
{
   ServiceHost host = new ServiceHost(typeof(MyService));
   host.Open();
   /* Some blocking operations */
   host.Close();
}

The thread pool will have three threads. Using the same service definition as in Figure 1, MyService will have an affinity to those threads. All calls to the service will be channeled to them, regardless of the service concurrency mode or instancing mode, and across all endpoints and contracts supported by the service.

The problem with the above code is that the service is at the mercy of the hosting code. What if, by design, the service is required to execute on the pool? It would be better to apply the thread pool declaratively, as part of the service definition. To that end, I wrote the ThreadPoolBehaviorAttribute:

[AttributeUsage(AttributeTargets.Class)]
public class ThreadPoolBehaviorAttribute : Attribute,
                                           IContractBehavior,
                                           IServiceBehavior
{
   public ThreadPoolBehaviorAttribute(uint poolSize,Type serviceType);
   public ThreadPoolBehaviorAttribute(uint poolSize,Type serviceType,
                                      string poolName);
}

You apply the attribute directly on the service, while providing the service type as a constructor parameter:

[ThreadPoolBehavior(3,typeof(MyService))]
class MyService : IMyContract
{...}

The attribute provides an instance of ThreadPoolSynchronizer to the dispatchers of the service's endpoints. The key in implementing ThreadPoolBehaviorAttribute is knowing how and when to hook up the dispatchers with the synchronization context. To facilitate this, ThreadPoolBehaviorAttribute supports the special WCF extensibility interface IContractBehavior:

public interface IContractBehavior
{
   void ApplyDispatchBehavior(ContractDescription description,
                              ServiceEndpoint endpoint,
                              DispatchRuntime dispatch);
   // More members
}

When a service is decorated with an attribute that supports IContractBehavior, after opening the host (but before forwarding calls to the service) and for each service endpoint, WCF calls the ApplyDispatchBehavior method, providing it with the DispatchRuntime parameter allowing it to affect the dispatcher. Figure 2 lists the implementation of ThreadPoolBehaviorAttribute.

Figure 2 Implementing ThreadPoolBehaviorAttribute (Close Figure 2)
[AttributeUsage(AttributeTargets.Class)]
public class ThreadPoolBehaviorAttribute : Attribute,IContractBehavior,
                                           IServiceBehavior
{
   //Store values in matching members
   protected string PoolName
   {get{...};set{...};}
   protected uint PoolSize
   {get{...};set{...};}
   protected Type ServiceType
   {get{...};set{...};}

   public ThreadPoolBehaviorAttribute(uint poolSize,Type serviceType)
: this(poolSize,serviceType,null)
   {}
   public ThreadPoolBehaviorAttribute(uint poolSize,Type serviceType,
       string poolName)
   {
      PoolName    = poolName;
      ServiceType = serviceType;
      PoolSize    = poolSize;
   }
   protected virtual ThreadPoolSynchronizer ProvideSynchronizer()
   {
      if(ThreadPoolHelper.HasSynchronizer(ServiceType) == false)
         return new ThreadPoolSynchronizer(PoolSize,PoolName);
      else
         return ThreadPoolHelper.GetSynchronizer(ServiceType);
   }

   void IContractBehavior.ApplyDispatchBehavior
(ContractDescription description,ServiceEndpoint endpoint,
            DispatchRuntime dispatch)
   {
      PoolName = PoolName ?? "Pool executing endpoints of " + ServiceType;

      ThreadPoolHelper.ApplyDispatchBehavior(ProvideSynchronizer(),
        PoolSize,ServiceType,PoolName,dispatchRuntime);
   }
   void IServiceBehavior.Validate(ServiceDescription description,
        ServiceHostBase serviceHostBase)
   {
      serviceHostBase.Closed += delegate
          {
              ThreadPoolHelper.CloseThreads(ServiceType);
          };
   }
   //Rest of the implementation
}
public static class ThreadPoolHelper
{
   static Dictionary<Type,ThreadPoolSynchronizer> m_Synchronizers =
       new Dictionary<Type,ThreadPoolSynchronizer>();

   [MethodImpl(MethodImplOptions.Synchronized)]
   internal static bool HasSynchronizer(Type type)
   {
      return m_Synchronizers.ContainsKey(type);
   }

   [MethodImpl(MethodImplOptions.Synchronized)]
   internal static ThreadPoolSynchronizer GetSynchronizer(Type type)
   {
      return m_Synchronizers[type];
   }
   [MethodImpl(MethodImplOptions.Synchronized)]
   internal static void ApplyDispatchBehavior(
ThreadPoolSynchronizer synchronizer,uint poolSize,Type type,
    string poolName,DispatchRuntime dispatch)
   {
      if(HasSynchronizer(type) == false)
      {
         m_Synchronizers[type] = synchronizer;
      }
      dispatch.SynchronizationContext = m_Synchronizers[type];
   }
   [MethodImpl(MethodImplOptions.Synchronized)]
   public static void CloseThreads(Type type)
   {
      if(m_Synchronizers.ContainsKey(type))
      {
         m_Synchronizers[type].Dispose();
         m_Synchronizers.Remove(type);
      }
   }
}

It is a best practice to separate the implementation of a WCF custom behavior attribute from the actual behavior. Let the attribute merely decide on the sequence of events and have a helper class provide the actual behavior. Doing so lets you use the behavior separately, for instance by a custom host. This is why ThreadPoolBehaviorAttribute does not do much—it delegates most of its work to a static helper class called ThreadPoolHelper. The ThreadPoolHelper class provides the HasSynchronizer method, which tells if the specified service type already has a synchronization context, and the GetSynchronizer method, which returns the synchronization context associated with the type. ThreadPoolBehaviorAttribute uses these two methods in the virtual ProvideSynchronizer method to ensure that it creates the pool exactly once for the service type. This check is required since ApplyDispatchBehavior can be called multiple times— once per endpoint, in fact. ThreadPoolBehaviorAttribute is also a custom service behavior because it implements IServiceBehavior:

public interface IServiceBehavior
{
   void Validate(ServiceDescription description,ServiceHostBase host);
   //More members
}

The Validate method of IServiceBehavior uses the service host instance to subscribe to the host Closed event, where it asks ThreadPoolHelper to terminate all the threads in the pool.

What ThreadPoolHelper does is associate all dispatchers of all endpoints of that service type with the same instance of ThreadPoolSynchronizer. This ensures that all calls are routed to the same pool. ThreadPoolHelper has to be able to map a service type to a particular ThreadPoolSynchronizer, so it declares a static dictionary called m_Synchronizers that uses service types as keys, and ThreadPoolSynchronizer instances as values.

In ApplyDispatchBehavior, ThreadPoolHelper checks to see if m_Synchronizers already contains the provided service type. If the type is not found, ThreadPoolHelper adds the provided ThreadPoolSynchronizer to m_Synchronizers, associating it with the service type. After that, it simply assigns the ThreadPoolSynchronizer instance to the dispatcher:

dispatch.SynchronizationContext = m_Synchronizers[type];

This single line is all that is required to have WCF use the custom synchronization context from now on. In the CloseThreads method, ThreadPoolHelper looks up from the dictionary the ThreadPoolSynchronizer instance and disposes of it (thus gracefully terminating all the worker threads in the pool). All access to the dictionary is automatically synchronized via the MethodImpl attribute and the MethodImplOptions.Synchronized flag. ThreadPoolHelper also verifies that the provided pool size value does not exceed the max concurrent calls value of the dispatcher throttle (not shown in Figure 2).

Thread Affinity Synchronization Context

A pool size of one will in effect create an affinity between a particular thread and all service calls, regardless of the service concurrency and instancing modes. This is particularly useful if the service is required to create some UI, such as a pop-up window, and periodically show, hide, and update it. Being the party creating the window, the service must enforce the fact it is being called on the same thread. Another use for thread affinity is a service that accesses or creates resources that use the TLS. To formalize such requirements, I create the specialized AffinitySynchronizer like so:

public class AffinitySynchronizer : ThreadPoolSynchronizer
{
   public AffinitySynchronizer() :
         this("AffinitySynchronizer Worker Thread")
   {}
   public AffinitySynchronizer(string threadName): base(1,threadName)
   {}
}

While you can install AffinitySynchronizer as in the earlier referenced code, I also defined the ThreadAffinityBehaviorAttribute in Figure 3. This attribute can be used like this:

Figure 3 ThreadAffinityBehaviorAttribute (Close Figure 3)
[AttributeUsage(AttributeTargets.Class)]
public class ThreadAffinityBehaviorAttribute :
                                       ThreadPoolBehaviorAttribute
{
   public ThreadAffinityBehaviorAttribute(Type serviceType)
          : this(serviceType,"Affinity Worker Thread")
   {}

   public ThreadAffinityBehaviorAttribute(
Type serviceType,string threadName):base(1,serviceType,
                                                   threadName)
   {}
}

[ThreadAffinityBehavior(typeof(MyService))]
class MyService : IMyContract
{...}

When relying on thread affinity, all service instances are always thread safe, since only a single thread (and the same thread at that) can access them. Configuring the service with ConcurrencyMode .Multiple amounts to creating single-threaded access. Also, be aware that configuring with ConcurrencyMode.Reentrant does not actually provide for reentrancy, since no reentrancy is possible while the single thread is blocked on the call out. It is best to select the default value of ConcurrencyMode.Single when relying on thread affinity.

Priority Processing

By default, all calls to your WCF service will be processed in the order in which they arrive. This is true both if you use the I/O Completion Ports thread pool or the custom thread pool. Normally, this is exactly what you want. But what if some calls have higher priority and you want to process them as soon as they arrive, rather then handling them in order? Even worse, what if the load on your service is such that the underlying service resources are exhausted? What if the throttle is maxed out? In these cases, your higher priority calls will be queued just as the other calls, waiting for the service or its resources to become available. Synchronization contexts offer an elegant solution to this problem: assign a priority to the calls, and have the synchronization context sort the calls as they arrive, before dispatching them to the thread pool for execution. This is exactly what my PrioritySynchronizer class (defined in Figure 4) does.

Figure 4 PrioritySynchronizer (Close Figure 4)
public enum CallPriority
{
   Low,
   Normal,
   High
}
public class PrioritySynchronizer : ThreadPoolSynchronizer
{
   public PrioritySynchronizer(uint poolSize);
   public PrioritySynchronizer(uint poolSize,string poolName);

   public static CallPriority Priority
   {get;set;}
}

PrioritySynchronizer derives from ThreadPoolSynchronizer and adds the sorting I just mentioned. Since the Send and Post methods of SynchronizationContext do not take a priority parameter, the client of PrioritySynchronizer has two ways of passing the priority of the call: the first is via the Priority property, which stores the priority (a value of the enum type CallPriority) in the TLS of the calling thread. The second option is via the message headers, which I will discuss later in this column. If left unspecified, Priority defaults to CallPriority.Normal.

I also provide the matching PriorityCallsBehaviorAttribute shown in Figure 5. Using PriorityCallsBehaviorAttribute is straightforward, as you can see here:

Figure 5 Implementing PriorityCallsBehaviorAttribute (Close Figure 5)
[AttributeUsage(AttributeTargets.Class)]
public class PriorityCallsBehaviorAttribute : ThreadPoolBehaviorAttribute
{
   public PriorityCallsBehaviorAttribute(uint poolSize,Type serviceType)
            : this(poolSize,serviceType,null)
   {}
   public PriorityCallsBehaviorAttribute(uint poolSize,Type serviceType,
       string poolName) : base(poolSize,serviceType,poolName)
   {}
   protected override ThreadPoolSynchronizer ProvideSynchronizer()
   {
      if(ThreadPoolHelper.HasSynchronizer(ServiceType) == false)
         return new PrioritySynchronizer(PoolSize,PoolName);
      else
         return ThreadPoolHelper.GetSynchronizer(ServiceType);
   }
}

[PriorityCallsBehavior(3,typeof(MyService))]
class MyService : IMyContract
{...}

PriorityCallsBehaviorAttribute overrides ProvideSynchronizer and provides PrioritySynchronizer instead of ThreadPoolSynchronizer. Because PrioritySynchronizer derives from ThreadPoolSynchronizer, this is transparent as far as ThreadPoolHelper is concerned

The real challenge in implementing and supporting priority processing is providing the call priority from the client to the service, and ultimately to PrioritySynchronizer. Using the Priority property of PrioritySynchronizer is only useful for non-WCF clients that interact directly with the synchronization context; it is of no use for a WCF client, whose thread is never used to access the service. While you could provide the priority as an explicit parameter in every method, I wanted a generic mechanism that could be applied on any contract and service. To that end, you have to pass the priority of the call out-of-band, via the message headers. In my February 2007 Foundations column called "Build a Queued WCF Response Service" (see msdn.microsoft.com/msdnmag/issues/07/02/Foundations), I faced a similar problem—passing the address of the response service out-of-band to the service—and I explain in detail the use of the incoming and outgoing headers. While I could use exactly the same technique here, it would be repetitive work. It is better to therefore extend the technique shown in that column, and augment WCF with general-purpose management of extraneous information from the client to the service, in effect, a generic yet type-safe and application-specific custom context. This is what the GenericContext<T> class provides. Figure 6 shows its implementation with error handling removed.

Figure 6 GenericContext<T> Class (Close Figure 6)
[DataContract]
public class GenericContext<T>
{
   [DataMember]
   public readonly T Value;

   public GenericContext(T value)
   {
      Value = value;
   }
   public GenericContext() : this(default(T))
   {}
   public static GenericContext<T> Current
   {
      get
      {
         OperationContext context = OperationContext.Current;
         if(context == null)
            return null;

      return context.IncomingMessageHeaders.GetHeader<GenericContext<T>>
         typeof(T).ToString(), typeof(T).Namespace);
      }
      set
      {
         OperationContext context = OperationContext.Current;
         MessageHeader<GenericContext<T>> genericHeader =
            new MessageHeader<GenericContext<T>>(value);
         context.OutgoingMessageHeaders.Add(genericHeader.GetUntypedHeader(
            typeof(T).ToString(), typeof(T).Namespace));
      }
   }
}

GenericContext<T> completely encapsulates the interaction with the message headers. All a client has to do to pass additional information to the service is set the static Current property inside a new OperationContextScope. For example, to pass the number 4 in the headers, use code as shown in Figure 7. You can further encapsulate GenericContext<T> via type-specific derivation such as IntContext shown in Figure 8.

Figure 8 Integer Context (Close Figure 8)
[DataContract]
public class IntContext : GenericContext<int>
{
   public IntContext()
   {}

   public IntContext(int number) : base(number)
   {}

   public static int Number
   {
      get
         return Current.Value;
      set
         Current = new IntContext(value);
   }
}
Figure 7 Using GenericContext<T> (Close Figure 7)
MyContractClient proxy = new MyContractClient();

using(OperationContextScope contextScope =
    new OperationContextScope(proxy.InnerChannel))
{
   GenericContext<int>.Current = new GenericContext<int>(4);
   proxy.MyMethod();
}

Using IntContext, Figure 7 would contain a line such as:

IntContext.Current = 4;

Literally any data contract (or serializable) type can be used for the type parameter in the custom context. On the service side, to read the value out of the custom headers, any downstream party can write:

int number = GenericContext<int>.Current.Value;

or using the type-specific custom context:

int number = IntContext.Number;

This is exactly what PrioritySynchronizer does when looking for the call priority. It expects the clients to provide the priority either in the TLS (via the Priority property) or in the form of a custom context that stores the priority in the message headers, using code similar to what you saw already in Figure 7:

GenericContext<CallPriority>.Current =
        new GenericContext<CallPriority>(CallPriority.High);
proxy.MyMethod();

However, I was not quite happy with such code because it is exposed, it couples the client to the priority mechanism, and it forces the client to set the priority every time before using the proxy. To deal with these issues, I created the HeaderClientBase<T,H> proxy base class shown in Figure 9.

Figure 9 HeaderClientBase<T,H> (Close Figure 9)
public abstract partial class HeaderClientBase<T,H> :
ClientBase<T> where T : class
{
   protected H Header;

   public HeaderClientBase() : this(default(H))
   {}

   public HeaderClientBase(string endpointName)
                                   : this(default(H),endpointName)
   {}

   public HeaderClientBase(H header)
   {
      Header = header;
   }

   public HeaderClientBase(H header,string endpointName)
                                    : base(endpointName)
   {
      Header = header;
   }

   /* More constructors */

   protected virtual object Invoke(string operation,params object[] args)
   {
      using(OperationContextScope contextScope =
           new OperationContextScope(InnerChannel))
      {
         GenericContext<H>.Current = new GenericContext<H>(Header);

         Type contract = typeof(T);
         MethodInfo methodInfo = contract.GetMethod(operation);
         return methodInfo.Invoke(Channel,args);
     }
   }
}

Like its base class, HeaderClientBase<T,H> accepts the contract type parameter T, but also the custom header type H. HeaderClientBase<T,H> uses the default value of H or a construction-time provided value to pass for all calls. Lacking a code generator support such as SvcUtil, I resorted to reflection to invoke the actual service operation, while completely encapsulating the interaction with the generic context and the operation context scope. You need to derive from HeaderClientBase<T,H> and provide your desired header type, as in Figure 10, providing an integer.

Figure 10 Using HeaderClientBase<T,H> (Close Figure 10)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   void MyMethod(string text);
}

class MyContractClient : HeaderClientBase<IMyContract,int>,IMyContract
{
   public MyContractClient(int number) : base(number)
   {}
   public void MyMethod(string text)
   {
      Invoke("MyMethod",text);
   }
}

By using the code in Figure 10, you can see that Figure 7 is reduced to merely:

MyContractClient proxy = new MyContractClient(4);
proxy.MyMethod();

While you could use the same technique as shown in Figure 10 to pass the call priority, for generality's sake it is better to define a general-purpose priority-enabled proxy, the PriorityClientBase<T>, shown in Figure 11.

Figure 11 Defining PriorityClientBase<T> (Close Figure 11)
public abstract partial class PriorityClientBase<T> :
     HeaderClientBase<T,CallPriority> where T : class
{
   public PriorityClientBase() : this(PrioritySynchronizer.Priority)
   {}

   public PriorityClientBase(string endpointName) :
       this(PrioritySynchronizer.Priority,endpointName)
   {}

   public PriorityClientBase(Binding binding,
EndpointAddress remoteAddress) :
       this(PrioritySynchronizer.Priority,binding,remoteAddress)
   {}

   public PriorityClientBase(CallPriority priority) : base(priority)
   {}

   public PriorityClientBase(CallPriority priority,string endpointName) :
       base(priority,endpointName)
   {}

   public PriorityClientBase(CallPriority priority,Binding binding,
       EndpointAddress remoteAddress) :
       base(priority,binding,remoteAddress)
   {}
   /* More constructors */
}

PriorityClientBase<T> hardcodes the use of CallPriority for the type parameter H. Also, PriorityClientBase<T> defaults to reading the priority from the TLS (yielding CallPriority.Normal when not found), so it can be used like any other proxy class. With very minor changes to your existing proxy classes, you can now add priority processing support, as shown in Figure 12.

Figure 12 Priority Processing Support (Close Figure 12)
class MyContractClient : PriorityClientBase<IMyContract>,IMyContract
{
   //Reads priority from TLS
   public MyContractClient()
   {}

   public MyContractClient(CallPriority priority) : base(priority)
   {}
   public void MyMethod(string text)
   {
      Invoke("MyMethod",text);
   }
}

MyContractClient proxy = new MyContractClient(CallPriority.High);
proxy.MyMethod();

Callbacks and Synchronization Context

When it comes to synchronization contexts, WCF offers similar support for callback objects with the CallbackBehaviorAttribute applied on the callback type:

[AttributeUsage(AttributeTargets.Class)]
public sealed class CallbackBehaviorAttribute : Attribute,...
{
   public bool UseSynchronizationContext
   {get;set;}
   //More members
}

As with the ServiceBehaviorAttribute, UseSynchronizationContext defaults to true. The interesting aspect of a callback object is when affinity to the synchronization context is locked in. WCF will examine the thread that opens the proxy that establishes the duplex communication. If that thread has a synchronization context, then that context will be used for all callbacks. While you could manually install a custom synchronization context by explicitly setting it before opening the proxy, it is better to do so declaratively using an attribute. To affect the callback endpoint dispatcher, the attribute needs to implement the IEndpointBehavior interface:

public interface IEndpointBehavior
{
   void ApplyClientBehavior(ServiceEndpoint endpoint,
                            ClientRuntime clientRuntime);
   //More members
}

In the ApplyClientBehavior method, the ClientRuntime parameter contains a reference to the endpoint dispatcher with the CallbackDispatchRuntime property:

public sealed class ClientRuntime
{
   public DispatchRuntime CallbackDispatchRuntime
   {get;}
   //More members
}

The rest is identical to the service-side attribute, as demonstrated by my CallbackThreadPoolBehaviorAttribute in Figure 13.

Figure 13 CallbackThreadPoolBehaviorAttribute (Close Figure 13)
[AttributeUsage(AttributeTargets.Class)]
public class CallbackThreadPoolBehaviorAttribute :
 ThreadPoolBehaviorAttribute,IEndpointBehavior
{
   public CallbackThreadPoolBehaviorAttribute(uint poolSize,
Type clientType) : this(poolSize,clientType,null)
   {}
   public CallbackThreadPoolBehaviorAttribute(uint poolSize,
Type clientType,string poolName) :
 base(poolSize,clientType,poolName)
   {
      AppDomain.CurrentDomain.ProcessExit += delegate
      {
         ThreadPoolHelper.CloseThreads(ServiceType);
      };
   }
   void IEndpointBehavior.ApplyClientBehavior
      (ServiceEndpoint serviceEndpoint,ClientRuntime clientRuntime)
   {
      IContractBehavior contractBehavior = this;
      contractBehavior.ApplyDispatchBehavior(null,serviceEndpoint,
         clientRuntime.CallbackDispatchRuntime);
   }
   //Rest of the implementation
}

In fact, in the callback attribute I wanted to reuse as much of the service attribute as possible. To that end, CallbackThreadPoolBehaviorAttribute derives from ThreadPoolBehaviorAttribute. Its constructors pass the client type to the base constructors as the service type. The CallbackThreadPoolBehaviorAttribute implementation of ApplyClientBehavior queries its base class for IContractBehavior (this is how a subclass uses an explicit private interface implementation of its base class) and delegates the implementation to ApplyDispatchBehavior.

The big difference between a client callback attribute and a service attribute is that the callback scenario has no host object to subscribe to its Closed event. To compensate, CallbackThreadPoolBehaviorAttribute monitors the process exit event to close all the threads in the pool. If the client wants to expedite closing those threads, the client can use ThreadPoolBehavior.CloseThreads, as shown in Figure 14.

Figure 14 Using CallbackThreadPoolBehaviorAttribute (Close Figure 14)
[ServiceContract(CallbackContract = typeof(IMyContractCallback))]
interface IMyContract
{
   [OperationContract]
   void MyMethod();
}
interface IMyContractCallback
{
   [OperationContract]
   void OnCallback();
}

[CallbackThreadPoolBehavior(3,typeof(MyClient))]
class MyClient : IMyContractCallback,IDisposable
{
   MyContractClient m_Proxy;

   public void CallService()
   {
      m_Proxy = new MyContractClient(new InstanceContext(this));

      m_Proxy.MyMethod();
   }

   //Called by threads from the custom pool
   public void OnCallback()
   {...}

   public void Dispose()
   {
      m_Proxy.Close();
      ThreadPoolHelper.CloseThreads(typeof(MyClient));
   }
}

Just like the service side, to have all the callbacks execute on the same thread (perhaps to create some UI on the callback side), you can configure the callbacks to have a pool size of 1. Better yet, you can define a dedicated callback attribute for such same-thread affinity, as in Figure 15.

Figure 15 CallbackThreadAffinityBehaviorAttribute (Close Figure 15)
[AttributeUsage(AttributeTargets.Class)]
public class CallbackThreadAffinityBehaviorAttribute :
                     CallbackThreadPoolBehaviorAttribute
{
   public CallbackThreadAffinityBehaviorAttribute(Type clientType) :
        this(clientType,"Callback Worker Thread")
   {}
   public CallbackThreadAffinityBehaviorAttribute(Type clientType,
        string threadName) : base(1,clientType,threadName)
   {}
}

Why Use Synchronization Contexts?

Synchronization contexts in the .NET Framework 2.0 are a powerful construct, and coupled with WCF, they yield a superior and productive programming model. The extensibility inherent to WCF makes this combination especially powerful, accomplishing difficult tasks such as thread affinity and priority processing with relatively few lines of code, and all outside the scope of your service. The custom headers and generic context techniques I demonstrated in this column are useful across the landscape of WCF programming. Wherever out-of-band parameters are called for, they provide an elegant, concise and type-safe solution. I also wanted to share with you my thought processes and the trade-offs to consider when evolving your own custom WCF extensions.

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a Software Architect with IDesign providing WCF training and WCF architecture consulting. His recent book is Programming WCF Services (O'Reilly, 2007). He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


Bugslayer: Measuring the Impact of View State

Bugslayer
Measuring the Impact of View State
John Robbins

Code download available at: Bugslayer2007_11.exe (209KB)
Contents
View State
Debugging Hints
Goals
Tracing View State
Implementation Highlights
Further Ideas

I
sn't it funny how the Microsoft® .NET Framework is thought of as an environment where you don't have to think about memory? Of course, what's one of the number one issues that continues to plague managed applications today? Memory! Why? When the garbage collector runs to reclaim memory, the common language runtime (CLR) suspends all the threads in the application—and no work gets done. When you're not getting any work done, you have a performance problem.

No matter how much you care about your .NET Framework memory usage, a couple of major problems interfere with actually getting a useful picture of that memory. The first is that though the .NET Framework does a tremendous amount of work on your behalf, the documentation is not necessarily helpful in figuring out what work is being performed and how much memory is being used. Fortunately, Lutz Roeder's Reflector for .NET (aisto.com/roeder/dotnet) helps considerably.

The second problem with observing memory usage is that the main development tool for .NET, Visual Studio®, does not show you anything about memory. That's not the fault of the Visual Studio team, but a failing in the original CLR architecture, which doesn't expose a means through the CLR Debugging API to see what generation a particular object is in. While it's possible to use the CLR Profiling API to keep track of object generations, it takes tremendous effort and does not work when you're debugging. As I wrote in the June 2003 (msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer) and again in the March 2005 (msdn.microsoft.com/msdnmag/issues/05/03/Bugslayer) Bugslayer columns, you can use the Son of Strike (SOS) WinDBG extension to see all the memory information you want, but it's probably the most painful tool to use since the original Windows® 95 kernel debugger.

In this episode of Bugslayer, I want to present a tool that lets you look at one of the most insidious performance killers in an ASP.NET app: view state. As you'll see, being able to see your view state usage easily on a production server can mean the difference between thirty minutes of debugging and six weeks of guessing.

View State

In an ASP.NET page, view state is an extremely nifty mechanism whereby the state of the controls on the page is stored in a hidden value within the HTML code sent to the client. This avoids maintaining the state of the page in the ASP.NET worker process, which would put even more memory pressure on the server. If you ever worked on an ISAPI filter written in native C++ back in the Jurassic period of the Internet (1996 or so), you remember how much effort went into keeping track of page, session, and application state. If you don't have much ASP.NET experience, you'll want read Dino Esposito's February 2003 Cutting Edge column on view state as a primer (msdn.microsoft.com/msdnmag/issues/03/02/cuttingedge).

The problem with view state is that you really have no idea how big that view state value is at any given time. This is especially true when there's a GridView control on the page. The ease of use in getting that database information on the page comes at a cost. The good news is that in ASP.NET 2.0, the Microsoft developers worked hard to bring the size of the view state down compared to ASP.NET 1.1. However, in the last couple of months I've helped debug several large ASP.NET 2.0 applications where the performance issues were caused by jumbo-size view states on certain pages.

Debugging Hints

Before we get into the tool and code, I want to discuss a few of my debugging tricks. When developers call me about an ASP.NET application that is having a performance problem, they invariably say, "The application was working fine, but now it is really slow."

My first question is, "Are you using any GridViews in your application?" As most ASP.NET applications are front ends to databases, that's pretty much a given, but not always. If they are not using GridViews, I need to look at a different path.

The second question I ask is, "Can I look at your database stored procedure history in version control?" Sadly, at this point, far too many development teams gulp hard and tell me they don't have their stored procedures in version control. The last time I looked, a SQL stored procedure was code, and if you follow the common-sense rules of development, all code goes into version control. In fact, Microsoft makes it exceedingly easy with the Visual Studio 2005 Team Edition for Database Professionals. If you don't have your stored procedures in version control, you need to stop reading right now and add them.

The reason I ask to look at their stored procedures in version control is that when the application is running fine and suddenly starts having issues, I need to find the changes between the previous and current editions. I can't begin to tell you how many times I've looked through the version control change logs for the stored procedures and seen where someone has changed a query to be something along the lines of:

   select * from table

When you're pumping that output to a GridView control (which relies heavily on the view state) and the database contains six million records, you have a serious view state problem!

Though view state is easy to use, it's nearly impossible to see when you've had a problem. Most developers know you can see the view state for a page by turning on ASP.NET tracing. Unfortunately, ASP.NET tracing only stores a limited number of traces and it can be extremely slow on a production server. Your alternative was to have each user right-click in the browser, select View Source from the context menu, and start manually counting characters. Clearly, something better was needed: a simple way to keep track of the largest view state for all pages on a production server so you could have your data in one place to make intelligent debugging decisions.

Goals

When I first started thinking about the challenges of a tool that lets you easily see your view state usage, the prime goal was that whatever I came up with had to be usable in a production environment. That meant the code had to be both fast and frugal with memory. Therefore, I made the decision to store only the largest view state size for each particular page. If I kept all of the view state sizes for every page, I could easily be forced to store millions of records. By storing only the largest view state for a given page, I drastically cut down the storage requirement, ensuring that there are never more data entries than there are number of .aspx files in the ASP.NET application.

I also wanted to make looking at the view state data simple and understandable, even for non-developers. While seeing the total view state size for a page is great, an additional requirement I added was an option to store and display the view state sizes for all of the controls on a page. That would make it much easier to identify the control that contributed most to a problem.

I wanted the option to see the view state size on HTTP GET requests as well. While nearly all view state problems are related to HTTP POST requests, users can store the full URL to a particular page and go directly to that page. That's a rare occurrence, but I wanted to include this option for completeness.

Finally, I wanted to ensure easy deployment for the tool. As a sub-goal, it would be ideal if my solution required no source code changes.

The source code download for this month's column includes the assembly that contains all the magic: Bugslayer.Web. Also included are the complete unit tests so you can play around with the assembly and see how it works. For production servers, you'll want to install Bugslayer.Web.DLL into the Global Assembly Cache (GAC), as that will let you load the code from anywhere on the machine.

There are two parts to Bugslayer.Web: an HttpModule and an HttpHandler. If you've read much about the ASP.NET processing pipeline, you can probably guess that the HttpModule is where the data collection occurs and the HttpHandler is what displays the data. And this means there are no code changes at all on your part to use Bugslayer.Web.

To configure your ASP.NET application to use the HttpModule, you'll need to add the <httpModules> element like the following to your web.config file:

<?xml version="1.0"?>
<configuration>
  <system.web>
    <httpModules>
      <add name="ViewStateStats"
          type="Bugslayer.Web.ViewStateStatisticsModule"/>
    </httpModules>
  </system.web>
</configuration>

Setting up the HttpHandler is nearly as easy. In your web.config file, add the <httpHandlers> element as follows:

<?xml version="1.0"?>
<configuration>
  <system.web>
    <httpHandlers>
      <add verb="*"
           path="viewstatestats.axd"
           type="Bugslayer.Web.ViewStateStatisticsHandler"
           validate="false" />
    </httpHandlers>
  </system.web>
</configuration>

Notice that the Bugslayer.Web.ViewStateStatisticsHandler is mapped to viewstatestats.axd. I chose to use the .axd extension because IIS already understands it as being an ASP.NET extension type. By setting the path attribute to a specific file name, ASP.NET will call that specific HttpHandler, but requests for other file names are routed to the ASP.NET internal .axd handler.

Tracing View State

After installing Bugslayer.Web and adding the entries to web.config, you'll be recording the largest view state byte count per page. While collecting the data is nice, you do need to be able to view it and, as discussed, that's what ViewStateStats.axd is for. Figure 1 shows example output. The format might look familiar as I borrowed the style from the ASP.NET tracing output. As you can see, the default is from largest to smallest view state. To sort on the URL or time, which both sort ascending, click on the column headers.

Figure 1 Example ViewStateStats.axd Output
Figure 1 Example ViewStateStats.axd Output(Click the image for a smaller view)
Figure 1 Example ViewStateStats.axd Output
Figure 1 Example ViewStateStats.axd Output (Click the image for a larger view)

You can control the view state data collection in the upper portion of ViewStateStats.axd page. The most interesting option is the second link, Collect Control View State Sizes. If the state is disabled (the default option) you'll only collect total sizes for the page. Clicking on the link will toggle the collection state to keep the list of child controls and their individual view states for the largest total view state for each page. While that will cause a performance hit and take more memory, there are plenty of times when you do need to figure out which control is hogging the view state. Figure 2 shows an example ViewStateStats.axd page with the controls expanded.

Figure 2 ViewStateStats.axd Control Tree Output
Figure 2 ViewStateStats.axd Control Tree Output(Click the image for a smaller view)
Figure 2 ViewStateStats.axd Control Tree Output
Figure 2 ViewStateStats.axd Control Tree Output (Click the image for a larger view)

Just like the ASP.NET tracing output, the control tree is displayed along with the name of the control and its view state size. If you're wondering where the render size bytes and ControlState size bytes values are, there's no way for me to get the render data, and I felt the ControlState size calculations would be too slow. I'll discuss why in the implementation section. However, since the view state is the major performance killer, the truly important control contribution is there. Note that the individual control sizes will not add up to the total view state size shown next to the URL and time. Since the view state is Base64 encoded, the encoding adds additional bytes to the overall view state.

The first link under the ViewStateStats.axd title, Clear all view state statistics, flushes all the existing saved values and starts fresh. The final link, Collect View State Sizes for HTTP GET Requests, turns on reporting for HTTP GET requests. As nearly every worry you have about view state is an HTTP POST, it's rare you'll need to turn on this option. However, if you do need to see view state sizes for both HTTP GETs and HTTP POSTs, you'll have the ability. If you toggle this option on, after each URL in the ViewStateStats.axd page you'll see a (GET) or (POST) as appropriate. When I discuss how I calculate the HTTP GET view state size, you'll see that there are performance implications for recording this data.

Implementation Highlights

When I first started toying around with the idea of recording the view state size for a whole application, I prototyped a derived System.Web.UI.Page class. It's trivial to get the view state string with the following C# code:

String viewState = Request.Params [ "__VIEWSTATE" ];

The problem with using a derived Page is that, for many existing ASP.NET applications, it would require you to change your code to see the view state across your application. Since HttpModules allow your code to participate in the normal ASP.NET pipeline, that was the perfect solution. Because I only wanted to inspect the page after it was ready to be sent down to the user, I set an EndRequest event handler in my IHttpModule.Init implementation method.

One interesting discovery that I made while reading about how the ASP.NET view state worked was following web.config setting:

<pages maxPageStateFieldLength="XX"/>

Here, XX is the number of maximum bytes for breaking up the __VIEWSTATE value into chunks. You can also set this with the System.Web.Ui.Page.MaxPageStateFieldLength property.

ASP.NET 2.0 introduced this setting in order to help break up huge __VIEWSTATE values because some firewalls and proxies choke on gigantic strings on the page. Given that gigantic strings are something you want to avoid, and the whole point of this column, you might think that setting the maxPageStateFieldLength could help reduce some size. Not only does it not help, it actually bloats your pages going down to the browser.

Say you have a view state like the following on the page:

<div>
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE"
value="/wEPDwULLTE1ODEyMjM1MTgPZBYCAgMPZBYCAgcPEGQPFgNmAgECAhYDEAUBYQUBYWcQBQFiBQFiZxAFAW
MFAWNnZGRkQIU7yEIZAT5uuif4cZcFHBJgZ48=" />
</div>

Setting maxPageStateFieldLength to 50 yields the following, which has far more actual text on the page:

<div>
<input type="hidden" name="__VIEWSTATEFIELDCOUNT"
       id="__VIEWSTATEFIELDCOUNT" value="3" />
<input type="hidden" name="__VIEWSTATE"
       id="__VIEWSTATE" value="/wEPDwULLTE1ODEyMjM1MTgPZBYCAgMPZBYCAgcPEGQPFgNmAg" />
<input type="hidden" name="__VIEWSTATE1"
       id="__VIEWSTATE1" value="ECAhYDEAUBYQUBYWcQBQFiBQFiZxAFAWMFAWNnZGRkQIU7yEIZ" />
<input type="hidden" name="__VIEWSTATE2"
       id="__VIEWSTATE2" value="AT5uuif4cZcFHBJgZ48=" />
</div>

While there are probably a few readers who have set maxPageStateFieldLength, I doubt many of you are using it. If you are breaking the view state into chunks to get around firewall and proxy issues, I strongly suggest you consider redesigning the page or storing your view state on the server to avoid sending huge amounts of data over the wire.

My HttpModule, which is implemented in ViewStateStatisticsModule.cs, properly handles the case where maxPageStateFieldLength has been set. Since I have to enumerate each of the __VIEWSTATE* parameters, the code I use is a little slower than accessing a single __VIEWSTATE parameter.

Because the data collection and display are logically coupled, I decided to group them behind the IViewStateProcessor interface, and then decouple all of that functionality from my HttpModule and HttpHandler. The IViewStateProcessor interface provides methods to process both HTTP GET and HTTP POST requests, and to display the data. My idea was to make it easy to provide additional data collection and display without having to touch most of the code in Bugslayer.Web.dll.

The class derived from IViewStateProcessor must be in memory at all times, so I store it in a static field inside my ViewStateStatisticsModule class. The ViewStateStatisticsHandler class simply accesses a static property in ViewStateStatisticsModule. Right now the code supports only a single IViewStateProcessor derived class. A nice addition would be to add support for simultaneously using additional classes.

To specify your own custom IViewStateProcessor-derived class instead, you'll need to override the HttpApplication.Init method. In there you'll access the ViewStateStatisticsModule through HttpApplication.Modules["ViewStateStats"]. Create an instance of your custom IViewStateProcessor-derived class and set the ViewStateStatisticsModule.ViewStateProcessor to your instance.

The default IViewStateProcessor-derived class, BiggestPageProcessor, is where all the real work occurs for collecting and displaying the data you saw inFigures 1 and 2. The collection data is stored in a Dictionary class with the URL itself as the key for fast lookup. The interesting part of the collection code was figuring out how to get the view states for individual controls.

As everyone is used to seeing the view state sizes for each individual control, I wanted to ensure my numbers matched for correctness. That meant I needed to figure out where in the ASP.NET tracing code the action happened. The first stop on the Reflector party boat was System.Web.Handlers.TraceHandler, which is the HttpHandler registered for trace.axd. About 15 minutes of exploring the decompiled code and looking for the referenced data types got me pointed at System.Web.Ui.Page.ProcessRequestMain, which performs the actual tracing if Page.Context.TracingEnabled is true. After bouncing through Page.BuildPageProfileTree, I finally got to Control.BuildProfileTree, which is where all the action happens to get the controls and their data into the trace output. Figure 3 shows the decompiled code.

Figure 3 Decompiled Control.BuildProfileTree Method (Close Figure 3)
protected void BuildProfileTree(string parentId, bool calcViewState)
{
    int viewStateSize;
    calcViewState = calcViewState && !this.flags[4];
    if (calcViewState)
    {
        viewStateSize = this.EstimateStateSize(this.SaveViewState());
    }
    else
    {
        viewStateSize = 0;
    }
    int controlStateSize = 0;
    if (((this.Page != null) &&
         (this.Page._registeredControlsRequiringControlState != null)) &&
         this.Page._registeredControlsRequiringControlState.Contains(
         this))
    {
        controlStateSize =
            this.EstimateStateSize(this.SaveControlStateInternal());
    }
    this.Page.Trace.AddNewControl(this.UniqueID, parentId,
        base.GetType().FullName,  viewStateSize, controlStateSize);
    if ((this._occasionalFields != null) &&
        (this._occasionalFields.Controls != null))
    {
        int count = this._occasionalFields.Controls.Count;
        for (int i = 0; i < count; i++)
        {
            this._occasionalFields.Controls[i].BuildProfileTree(
                this.UniqueID, calcViewState);
        }
    }
}

The interesting part of the code for view state is the first 10 lines of the method. The first bit of code that really threw me is the !this.flags[4] reference because Reflector shows the flags field as private. Using Reflector's lifesaving Analyzer, I jumped to the flags field and clicked on the "Used By" tree element. Scanning for anything related to view state showed that the Control.EnableViewState property getter was in the list. Hopping over to the decompiled EnableViewState getter method showed that the method actually returns !this.flags[4], so that mystery was solved.

The EstimateStateSize just creates an ObjectStateFormatter, calls the Serialize method, and returns the size of the serialization bytes, so there was no trouble duplicating that. Things are definitely more interesting with the call to the protected this.SaveViewState method and I wanted to call it from my HttpModule. Given that .NET reflection is the secret sauce to reach out and call anything, I used a dash of it to ensure that I was going to get the same numbers as the ASP.NET tracing system. Figure 4 shows the core code I wrote for calculating individual control view state sizes.

Figure 4 Core Control View State Size Calculation Code (Close Figure 4)
private PageViewState CreatePageViewState (
    String url ,
    DateTime timeStamp ,
    Int32 viewStateLength ,
    String httpRequestType )
{
    PageViewState returnValue = new PageViewState (
        url ,
        timeStamp ,
        viewStateLength ,
        httpRequestType );
    if ( true == collectControlData )
    {
        Page currentPage = HttpContext.Current.Handler as Page;
        if ( null != currentPage )
        {
            CalculateControlSizes ( currentPage ,
                                    0 ,
                                    returnValue.ControlList );
        }
    }
    return ( returnValue );
}

private static void CalculateControlSizes ( Control ctl ,
                                            Int32 indentLevel ,
                                            List<ControlViewState> ctlList )
{
    Int32 viewStateSize = EstimateSize ( ctl );
    ControlViewState state = new ControlViewState ( indentLevel ,
                                                ctl.UniqueID ,
                                                ctl.GetType ( ).FullName ,
                                                viewStateSize );
    // Add it to the list.
    ctlList.Add ( state );
    for ( int i = 0 ; i < ctl.Controls.Count ; i++ )
    {
        CalculateControlSizes ( ctl.Controls [ i ] , indentLevel + 1 ,
                                ctlList );
    }
}

private static Int32 EstimateSize ( Control ctl )
{
    // In Control.BuildProfileTree, the check is done against flag[4],
    // which is the same as the implementation of EnableViewState.
    if ( false == ctl.EnableViewState )
    {
        return ( 0 );
    }
    Object saved = CallSaveViewStateMethod ( ctl );
    if ( null == saved )
    {
        return ( 0 );
    }
    ObjectStateFormatter osf = new ObjectStateFormatter ( );
    Int32 ret = osf.Serialize ( saved ).Length;
    return ( ret );
}

private static Object CallSaveViewStateMethod ( Control ctl )
{
    // This is way ugly, but the SaveViewState method is protected on
    // the Control class so in order to get the total size, I need to
    // call it through Reflection.
    Object ret = null;
    BindingFlags flags = BindingFlags.Instance | BindingFlags.Public |
                            BindingFlags.NonPublic;
    Type type = ctl.GetType ( );
    MethodInfo m = type.GetMethod ( "SaveViewState" , flags );
    if ( null != m )
    {
        ret = m.Invoke ( ctl , null );
    }
    return ( ret );
}

private Int32 SumControlSizes ( Control ctl )
{
    Int32 viewStateSize = EstimateSize ( ctl );
    for ( int i = 0 ; i < ctl.Controls.Count ; i++ )
    {
        viewStateSize += SumControlSizes ( ctl.Controls [ i ] );
    }
    return ( viewStateSize );
}

Once I figured out the trick to calculating the view state just like the ASP.NET tracing code, I then needed to start testing to see whether the view state values were the same when I was asking for them inside the HttpModule.EndRequest event handler. The good news is that, after running all sorts of pages through Bugslayer.Web, I never saw a discrepancy between what tracing reported and what I calculated.

If you're curious why I didn't include render size bytes in the display, it's because there's no way for me to calculate those values. That happens during page rendering and my HttpModule is far too late in the pipeline. As for the ControlState size bytes, I chose not to calculate them for performance reasons. Figure 3 shows the work necessary to do the math. However, I would have had to use lots of slow reflection to make those calls for very little payback.

Eagle-eyed readers might have noticed that I discussed how getting the view state through Request.Params["__VIEWSTATE"] is accessing a form variable that does not exist with an HTTP GET request. However, if you look at the source code for a page in the browser, the __VIEWSTATE variable is certainly there, so I wanted to find a way to get the data in case a user goes to a page directly in the browser address bar.

After all sorts of poking around, the best solution I could come up with was to do the calculations myself by looping through the controls and getting each individual view state size. While not ideal, it does get close to the actual view state size without getting into full BASE64 encoding and the additional overhead. Because of this limitation on HTTP GET processing, I left the option for monitoring it off by default.

Further Ideas

Overall, I'm quite happy how the view state monitoring tool turned out. By making it very easy to integrate into an existing production application, my goal of looking for those fat view states should become a good bit easier to realize. When you've got a silent performance killer in the view state, being able to look at the page view state sizes easily means you'll be debugging faster so you can work on new features and the fun parts of why we all got in this business.

As with all Bugslayer columns, there are always additional features you can add to Bugslayer.Web to track even more information about your view state situations.

The version of the tool available with this column only tracks view state that's on the page in the form of a __VIEWSTATE hidden field because that's the worst performance killer of all. You may want to consider extending Bugslayer.Web to track view state information stored on the server. That will require more work as you'll need to provide a custom Page-derived class to override the Page.SaveViewState method.

In the sample, the view state storage is only on a per HttpApplication basis. A nice addition would be to track the largest view state across a Web farm.

As I wanted to make the control tree like what ASP.NET tracing shows, I'm including controls that have no view state data. A quick optimization would be to collect only those controls that contribute to view state. Better yet, make it a toggle-able option in the ViewStateStats.axd handler.

If you're a real HTML and JavaScript wizard, you could certainly make the ViewStateStats.axd page display better. I chose the simplest form of display, but having expandable and collapsible control display would make it easier to see very large Web site data.

Finally, on a personal note, I would dearly like to thank all the readers for reading 10 years of Bugslayer columns. All your questions and comments over the years have made writing these columns far more fun than you can ever imagine!

Tip 80 Recently, I was working on a problem where an application pool in IIS was recycling unexpectedly. In my desperate Internet searching I ran across a great blog entry from Johan Straarup, "Common reasons why your application pool may unexpectedly recycle" at blogs.msdn.com/johan/archive/2007/05/16/common-reasons-why-your-application-pool-may-unexpectedly-recycle.aspx. It's a great summary of all the reasons you're wondering about when W3WP.EXE has gone for a walk in the park. Johan's blog is one you need to subscribe to.

Tip 81 Want to become a better debugger? Roberto Farah, who has forgotten far more about WinDBG than any of us could hope to know, put together one of the best lists of debugging books I have ever seen at blogs.msdn.com/debuggingtoolbox/archive/2007/06/08/recommended-books-how-to-acquire-or-improve-debugging-skills.aspx. No matter what type of development you're doing on Windows, Roberto has you covered. In his blog, Roberto is doing things with WinDBG scripting that will completely blow your mind! My personal favorite is his WinDBG script to cheat at Mine Sweeper (blogs.msdn.com/debuggingtoolbox/archive/2007/03/28/windbg-script-playing-with-minesweeper.aspx). Now I win every time!

Tip 82 If you've read past Bugslayer columns, you know I'm a freak about MSBuild tasks (msdn.microsoft.com/msdnmag/issues/06/03/Bugslayer). You should automate the world through MSBuild. The SDC Tasks have been updated to include over 300 tasks, including creating Web sites, application pools, and Active Directory users, running FxCop, configuring virtual servers, creating zip files, configuring COM+, creating folder shares, installing into the GAC, configuring SQL Server, and configuring BizTalk® Server 2004 and BizTalk Server 2006. To control life as you know it from an automated build, download the new tasks from codeplex.com/sdctasks.

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in the .NET and Windows platforms. His latest book is Debugging Microsoft .NET 2.0 Applications (Microsoft Press, 2006). You can contact John at wintellect.com.


Concurrent Affairs: Simplified APM with C#

Concurrent Affairs
Simplified APM with C#
Jeffrey Richter

Contents
Anonymous Methods and Lambda Expressions
The foreach Statement
Iterators
Asynchronous Programming

I
have long extolled the virtues of the asynchronous programming model (APM), emphasizing that performing I/O-bound operations asynchronously is the key to producing scalable and responsive applications and components. Achieving these goals is possible because the APM allows you to use a very small number of threads to execute a large amount of work without blocking any of the threads. Unfortunately, using the APM to build a typical application or component is difficult, so many programmers just don't do it.

There are several things that make the APM difficult to implement. First, you need to avoid having state data reside on the thread's stack because data on the thread's stack cannot migrate to another thread. Avoiding stack-based data means having to avoid method arguments and local variables. Over the years, developers have come to adore arguments and local variables, as they make programming so much easier.

Second, you need to split up your code into callback methods, sometimes called continuations. For example, in one method you initiate an asynchronous read or write operation, and then you have to implement another method that will get called (possibly by a different thread) to process the result of the I/O operation. Programmers are just not used to thinking about data processing in this manner. Migrating state across methods and threads means that the state must be packaged up, which ends up complicating the implementation.

Third, there are many useful programming constructs—such as try/catch/finally statements, the lock statement, the using statement, and even loops (for, while, and foreach statements)—that can't be split up across multiple methods. Avoiding these constructs also complicates the implementation.

Finally, trying to offer features such as coordinating the result of multiple overlapping operations, cancellation, timeout, and marshaling UI control modifications to the GUI thread in a Windows® Forms or Windows Presentation Foundation (WPF) application adds even more complexity to using the APM.

In this column I will demonstrate some relatively recent additions to the C# programming language that can be used to make working with the APM significantly easier. Then I'll introduce a class of my own, called AsyncEnumerator, that builds on these C# language features in order to solve the problems I've just mentioned. My AsyncEnumerator class makes it easy and fun to use the APM in your code. With this class, you should have no reason not to use asynchronous programming, enabling your code to be scalable and responsive. Please note that the AsyncEnumerator class is part of my Power Threading library and relies on other code that is part of this library; the library is downloadable from Wintellect.com.

Anonymous Methods and Lambda Expressions

In Figure 1, the SynchronousPattern method shows how to open a file and read from it synchronously. The method is straightforward; it constructs a FileStream object, allocates a Byte[], calls Read, and then processes the returned data. The C# using statement ensures that the FileStream object is closed after processing of the data has completed.

Figure 1 Excerpts from the ApmPatterns Class (Close Figure 1)
private static void SynchronousPattern() {
    using (FileStream fs =
        new FileStream(typeof(ApmPatterns).Assembly.Location,
        FileMode.Open, FileAccess.Read, FileShare.Read, 8192,
        FileOptions.Asynchronous)) {
    Byte[] data = new Byte[fs.Length];
    Int32 bytesRead = fs.Read(data, 0, data.Length);
    ProcessData(data);
    }
}

#region APM Pattern with Multiple Methods
private sealed class ApmData {
    public FileStream m_fs;
    public Byte[] m_data;
}

private static void ApmPatternWithMultipleMethods() {
    ApmData apmData = new ApmData();
    apmData.m_fs =
        new FileStream(typeof(ApmPatterns).Assembly.Location,
        FileMode.Open, FileAccess.Read, FileShare.Read, 8192,
        FileOptions.Asynchronous);
    apmData.m_data = new Byte[apmData.m_fs.Length];
    apmData.m_fs.BeginRead(apmData.m_data, 0, apmData.m_data.Length,
        ReadCompleted, apmData);
    // Method returns after issuing async I/O request
}

private static void ReadCompleted(IAsyncResult result) {
    ApmData apmData = (ApmData) result.AsyncState;
    Int32 bytesRead = apmData.m_fs.EndRead(result);
    ProcessData(apmData.m_data);
    apmData.m_fs.Close(); // No 'using'
}
#endregion

private static void ApmPatternWithAnonymousMethod() {
    FileStream fs =
        new FileStream(typeof(ApmPatterns).Assembly.Location,
        FileMode.Open, FileAccess.Read, FileShare.Read, 8192,
        FileOptions.Asynchronous);
    Byte[] data = new Byte[fs.Length];
    fs.BeginRead(data, 0, data.Length, delegate(IAsyncResult result) {
        Int32 bytesRead = fs.EndRead(result);
        ProcessData(data);
        fs.Close(); // No 'using'
    }, null);
    // Method returns after issuing async I/O request
}

private static void ApmPatternWithLambdaExpression() {
    FileStream fs =
        new FileStream(typeof(ApmPatterns).Assembly.Location,
        FileMode.Open, FileAccess.Read, FileShare.Read, 8192,
        FileOptions.Asynchronous);
    Byte[] data = new Byte[fs.Length];
    fs.BeginRead(data, 0, data.Length, result => {
        Int32 bytesRead = fs.EndRead(result);
        ProcessData(data);
        fs.Close(); // No 'using'
    }, null);
    // Method returns after issuing async I/O request
}

The ApmPatternWithMultipleMethods method shows how to perform the same operation as the SynchronousPattern method using the APM of the common language runtime (CLR). Immediately, you'll see how much more complicated the implementation is. Notice that the ApmPatternWithMultipleMethods method initiates the asynchronous I/O operation and the ReadCompleted method gets called when the operation completes. Also note that passing data between the two methods is accomplished by packaging the shared data into an instance of the ApmData class, which I had to define specifically to enable the passing of this data between these two methods. You should also notice that the C# using statement cannot be used because the FileStream is opened in one method and is then closed in a different method. To compensate for this, I had to write code to explicitly call FileStream's Close method just before the ReadCompleted method returns.

The ApmPatternWithAnonymousMethod method demonstrates how this code could be rewritten using a new C# 2.0 feature called anonymous methods, which allows you to pass code as an argument to a method. It effectively allows you to embed one method's code inside another method's code. (I describe anonymous methods in great detail in my book CLR via C#, Microsoft Press, 2006.) Notice that the ApmPatternWithAnonymousMethod method is much shorter and easier to understand—after you get used to working with anonymous methods.

First, notice that the code is simpler because it is all contained inside just one method. In this code, I am calling the BeginRead method to initiate an asynchronous I/O operation. All BeginXxx methods take for their second-to-last parameter an AsyncCallback that is a delegate referring to a method that should be called by a thread pool thread when the operation completes. Normally, when using the APM, you'd have to write a separate method, give that method a name, and pass additional data to the method via the BeginXxx method's last argument. However, the anonymous method feature allows me to just write the separate method inline so all the code to initiate the request and process the result is in context. In fact, the code looks somewhat similar to the SynchronousPattern method.

Second, notice that the ApmData class is no longer necessary; you don't have to define it, construct an instance of it, and work with any of its fields! How is this possible? Well, anonymous methods do more than just let you embed one method's code inside another method's code. When the C# compiler detects any arguments or local variables declared in the outer method that are also used in the inner method, the compiler actually defines a class for you automatically, and each variable shared between the two methods becomes a field in this compiler-defined class. Then, inside the ApmPatternWithAnonymousMethod method, the compiler generates code to construct an instance of this class, and any code that refers to the variables is compiled into code that accesses the fields of the compiler-defined class. The compiler also makes the inner method an instance method on the new class, which allows its code to access the fields easily and now the two methods can share the data.

This is an awesome feature of anonymous methods as it allows you to write code that looks like you're using method arguments and local variables, but the compiler is actually rewriting your code, taking those variables off the stack, and embedding them as fields in an object. The object can easily be passed between methods and can easily migrate from one thread to another, which is just perfect when using the APM. And since the compiler is doing all this work automatically, you can simply pass null for the last parameter to the BeginRead method since there is now no data that has to be passed explicitly between the methods and threads. I still cannot use the C# using statement, however, as there are still two different methods here even though it looks like just one method.

The following shows the output from executing the code excerpted in Figure 1:

Primary ThreadId=1
ThreadId=1: 4D-5A-90-00-03 (SynchronousPattern)
ThreadId=3: 4D-5A-90-00-03 (ApmPatternWithMultipleMethods)
ThreadId=3: 4D-5A-90-00-03 (ApmPatternWithAnonymousMethod)
ThreadId=3: 4D-5A-90-00-03 (ApmPatternWithLambdaExpression)
ThreadId=3: 4D-5A-90-00-03 (ApmPatternWithIterator)

I had the Main method display the managed thread Id of the application's primary thread. Then I have the ProcessData method display the managed thread ID of the thread that is executing it. As you can see, the output shows that all the asynchronous patterns have the results executed by a thread other than the primary thread, while the synchronous pattern has everything executing via the application's primary thread.

I should also point out that C# 3.0 introduces a new feature called lambda expressions. The lambda expression feature is a more succinct syntax for doing the same thing as the C# anonymous method feature. In fact, this is a hassle for the C# team since it now has to document and support two different syntaxes that produce identical results. Modifying the ApmPatternWithAnonymousMethod method to use the C# 3.0 lambda expression feature gives you the ApmPatternWithLambdaExpression method shown in Figure 1. Here you see that the syntax is slightly simplified as the compiler is able to infer the type of the result argument as IAsyncResult automatically and "=>" is less typing than "delegate."

The foreach Statement

C# 2.0 introduces another feature into the C# programming language: iterators. The original intent of the iterator feature was to make it easy for developers to write code to traverse a collection's internal data structure. To understand iterators, you must first take a good look at the C# foreach statement. When you write code such as the following, the compiler turns it into what you see in Figure 2:

Figure 2 foreach Loop in C# (Close Figure 2)
// Construct an object that knows how to traverse the
// internal data structure of the collection
IEnumerator<String> e = collectionObject.GetEnumerator();
try {
   String s;
   while (e.MoveNext()) { // Advance to the next item
      s = e.Current;      // Get the item to process
      DoSomething(s);
   }
}
finally {
   // If the IEnumerable implements IDisposable,
   // clean up the object
   IDisposable d = e as IDisposable;
   if (d != null) d.Dispose();
}

foreach (String s in collectionObject)
   DoSomething(s);

You see, the foreach statement gives you an easy way to iterate over all the items in a collection class. However, there are lots of different kinds of collection classes—arrays, lists, dictionaries, trees, linked-lists, and so on—and each uses its own internal data structure for representing the collection of items. When you use the foreach statement, you're saying that you want to iterate over all the items in the collection and you don't care what data structure the collection class uses internally to maintain its items.

The foreach statement causes the compiler to generate a call to the collection class's GetEnumerator method. This method creates an object that implements the IEnumerator<T> interface. This object knows how to traverse the collection's internal data structure. Each iteration of the while loop calls the IEnumerator<T> object's MoveNext method. This method tells the object to advance to the next item in the collection and return true if successful or false if all items have been enumerated. Internally, MoveNext might just increment an index, or it might advance to the next node of a linked-list, or it might traverse up or down through a tree. The whole point of the MoveNext method is to abstract away the collection's internal data structures from the code that is executing the foreach loop.

If MoveNext returns true, then the call to the Current property's get accessor method returns the value of the item that the MoveNext method just advanced to so that the body of the foreach statement (the code inside the try block) can process the item. When MoveNext determines that all of the collection's items have been processed, it returns false.

At this point, the while loop exits and the finally block is entered. In the finally block, a check is made to determine whether the IEnumerator<T> object implements the IDisposable interface and, if so, its Dispose method is called. For the record, the IEnumerator<T> interface derives from IDisposable, thereby requiring all IEnumerator<T> objects to implement the Dispose method. Some enumerator objects require additional resources while iterating. Perhaps the object returns lines of text from a text file. When the foreach loop exits (which could happen before iterating though all the items in the collection), then the finally block calls Dispose, allowing the IEnumerator<T> object to release these additional resources—closing the text file, for example.

Iterators

Now that you see what code the foreach statement produces, I'll take a look at how an object implements the IEnumerable<T> interface along with its MoveNext and Current members. Let's talk about arrays since they are probably the simplest of all collection data structures. The System.Array class, which is the base class of all arrays, implements the IEnumerable<T> interface. Its GetEnumerator method returns an object whose type implements the IEnumerator<T> interface. So you can easily write code to iterate through all of an array's elements:

Int32[] numbers = new Int32[] { 1, 2, 3 };
foreach (Int32 num in numbers)
   Console.WriteLine(num);

As a side note, if the C# compiler detects that the collection used with foreach is an array, the compiler produces optimized code that has a simple index that advances from 0 to the length of the array minus one. But, for this discussion, let's ignore that the compiler performs this optimization and pretend that the compiler treats arrays just as it does any other collection class. In fact, you can force the compiler to treat an array just like any other collection class by casting it to IEnumerable<T> as follows:

foreach (Int32 num in (IEnumerable<Int32>) numbers)
   Console.WriteLine(num);

My ApmPatterns class includes an ArrayEnumerator<T> class that knows how to return each of the items in an array in reverse order (see Figure 3). This class implements the IEnumerator<T> interface, which requires that the class also implement the non-generic IEnumerable and IDisposable interfaces as well. The AsyncEnumerator<T> class offers a public constructor that saves the passed array in a private member. There is another private member, m_index, that indicates which item from the array you are currently up to iterating; this field is initialized to the length of the array (the element past the last element in the array). You can see this in Figure 3.

Figure 3 Iterators (Close Figure 3)
private static void Iterators() {
    Int32[] numbers = new Int32[] { 1, 2, 3 };

    // Use Array's built-in IEnumerable
    foreach (Int32 num in numbers)
    Console.WriteLine(num);

    // Use my ArrayEnumerable class
    foreach (Int32 num in new ArrayEnumerable<Int32>(numbers))
    Console.WriteLine(num);

    // Use my iterator
    foreach (Int32 num in ArrayIterator(numbers))
    Console.WriteLine(num);
}

private sealed class ArrayEnumerator<T>: IEnumerator<T> {
    private T[] m_array;
    private Int32 m_index;

    public ArrayEnumerator(T[] array) {
        m_array = array; m_index = array.Length;
    }
    public Boolean MoveNext() {
        if (m_index == 0) return false;
        m_index--; return true;
    }
    public T Current { get { return m_array[m_index]; } }
    Object IEnumerator.Current { get { return Current; } }
    public void Dispose() { }
    public void Reset() { }
}

private sealed class ArrayEnumerable<T>: IEnumerable<T> {
    private T[] m_array;
    public ArrayEnumerable(T[] array) { m_array = array; }
    public IEnumerator<T> GetEnumerator() {
        return new ArrayEnumerator<T>(m_array);
    }
    IEnumerator IEnumerable.GetEnumerator() {
        return GetEnumerator();
    }
}

private static IEnumerable<T> ArrayIterator<T>(T[] array) {
    try { yield return default(T); }
    finally {  }
    for (Int32 index = array.Length - 1; index >= 0; index--)
    yield return array[index];
}

private static IEnumerator<Int32>
    ApmPatternWithIterator(AsyncEnumerator ae) {

    using (FileStream fs =
        new FileStream(typeof(ApmPatterns).Assembly.Location,
        FileMode.Open, FileAccess.Read, FileShare.Read, 8192,
        FileOptions.Asynchronous)) {

        Byte[] data = new Byte[fs.Length];
        fs.BeginRead(data, 0, data.Length, ae.End(), null);
        yield return 1;

        Int32 bytesRead = fs.EndRead(ae.DequeueAsyncResult());
        ProcessData(data);
    }
}

All IEnumerator<T> objects are essentially state machines that keep track of which item they are up to enumerating from the collection. When the foreach loop calls MoveNext, the MoveNext method advances the state machine to the next item. In my ArrayEnumerator class, MoveNext returns false if all items have been enumerated (m_index is already at 0, the last item). If that is not the case, it decrements the m_index field and returns true. The Current property simply returns the item reflected by the current position of the state machine.

As you can see, the code is tedious to implement and I left some of the interface methods empty in order to simplify the coding. I also left out some error-checking code as well. Also, an array is a pretty simple data structure and so writing an enumerator for it is fairly straightforward. Other data structures, such as a linked list or tree, would significantly complicate the code.

One more thing that I should point out is that you cannot use my ArrayEnumerator<T> class directly with the foreach statement. In other words, this won't compile:

foreach (Int32 num in new ArrayEnumerator<Int32>(numbers))
   Console.WriteLine(num);

The foreach statement expects a collection (an object whose type implements IEnumerable<T>) after the in keyword, not an IEnumerator<T>. So to use my ArrayEnumerator<T>, I had to also define a class that implements IEnumerable<T>, my ArrayEnumerator<T> class also shown in Figure 3. Now, with the ArrayEnumerable<T> class defined, I can write the following code, which will compile and execute just fine:

foreach (Int32 num in new ArrayEnumerable<Int32>(numbers))
   Console.WriteLine(num);

Wow! This is quite a bit of code to write in order to iterate through a collection of items. If only there were some technology available to us that could make all of this code simpler to write. Fortunately, there is—the C# 2.0 iterator feature. An iterator allows you to write a single member that returns an ordered sequence of values. You get to write this using simple syntax to express how values should be returned and the C# compiler turns your block of code into a full-fledged class that implements the IEnumerable<T>, IEnumerable, IEnumerator<T>, IEnumerator, and IDisposable interfaces for you.

Using the C# iterator feature, I replace my AsyncEnumerable<T> class and AsyncEnumerator<T> classes with the following one member:

private static IEnumerable<T> ArrayIterator<T>(T[] array) {
   for (Int32 index = array.Length-1; index >= 0; index--)
      yield return array[index];
}

And, I can invoke this member with a foreach statement as follows:

foreach (Int32 item in ArrayIterator(numbers))
   Console.WriteLine(item);

As an added bonus, since this is a generic method (versus a generic class), C# type inference also kicks in and I do not have to write ArrayIterator<Int32>, making the foreach code simpler as well.

Note that, since my iterator member returns IEnumerable<T>, the compiler generates code that implements IEnumerable<T> and IEnumerable as well as IEnumerator<T>, IEnumerator, and IDisposable. However, you can write an iterator member that returns an IEnumerator<T>, in which case the compiler generates a class definition that implements only the IEnumerator<T>, IEnumerator, and IDisposable interface members.

I'll also point out that you can define an iterator member with a return type that is the non-generic IEnumerable or IEnumerator interface, then the compiler defines IEnumerable<Object>/ IEnumerable (if you specified IEnumerable as the return type), IEnumerator<Object>/IEnumerator, and IDisposable.

In an iterator, the yield return statement effectively instructs the compiler what value (collection item) to return from the method. However, an iterator doesn't actually end its execution at this point; instead, we say its execution is suspended. The next time MoveNext is called, the iterator resumes its execution with the statement immediately following the yield return statement. In addition to yield return, an iterator can contain a yield break statement that, when executed, causes the MoveNext method to return false, forcing the foreach loop to terminate. Exiting from an iterator member also causes false to be returned from the MoveNext method, forcing the foreach loop to terminate.

Asynchronous Programming

For asynchronous programming, iterators give you some really great benefits. First, within an iterator, you can write code that accesses arguments and local variables. However, the compiler is actually packing these variables up into class fields for you (just like it did for anonymous methods and lambda expressions).

Second, the compiler lets you write sequential code, but you have the ability to suspend the method midstream and continue its execution later (potentially using a different thread).

Third, you can use programming constructs such as try/catch/finally statements, lock statements, using statements, and even loops (for, while, and foreach statements) within an iterator. The compiler automatically re-writes your code so that these constructs maintain their semantics.

For the record, there are limitations associated with iterators:

  • An iterator member's signature cannot contain any out or ref parameters.
  • An iterator cannot contain a return statement (yield return is OK).
  • An iterator may not contain any unsafe code.
  • A finally block may not contain a yield return or a yield break statement.
  • A yield return statement may not appear in a try block that has a catch block (yield break is OK).
  • A yield return statement may not appear in a catch block (yield break is OK).

While it is good to keep these limitations in mind, for most programming scenarios, these limitations are either not an issue at all or can be worked around.

So now, armed with this knowledge of iterators, you could build an iterator that executes a series of asynchronous operations. In Figure 3, refer to the ApmPatternWithIterator member. This member is an iterator since the member's return type is an IEnumerator<T>. Notice how similar the code in this member is to the SynchronousPattern method discussed earlier. Specifically, notice how the iterator contains a using statement and how it uses an argument (ae) and local variables (fs and data).

You can tell that this member performs asynchronous operations because it calls BeginRead and EndRead (instead of just calling Read). But when using the APM, you normally call BeginRead and pass a callback method. In this case I am passing the result of calling ae.End (I'll explain this in my next column). Usually the method would return after calling BeginRead, but in an iterator I'm calling yield return, which just suspends the code.

Normally, when using the APM, the call to EndRead would be in a different method, but with iterators the call to EndRead is immediately following the yield return statement. This means that it will execute when the iterator's execution resumes. And another thread can execute this code. In fact, if you look at the output shown earlier, you see that the ApmPatternWithIterator's call to ProcessData executes via a different thread (ThreadId=3) than the application's primary thread (ThreadId=1).

If you really study this code carefully, you'll see that what's happening here is truly awesome and makes asynchronous programming almost as simple to execute as synchronous programming. But now your threads are never blocked, allowing your application to stay responsive to the user. Because threads are not blocked, there isn't a need to create more threads, so the application uses very few resources.

You can have multiple iterators executing simultaneously, which increases your application's scalability. And you can easily add the ability to cancel outstanding operations, report progress, set timers, coordinate the result of multiple overlapping operations, and have all of this execute on the GUI thread if updating the UI is desired. In addition, you can have all of this with a programming model that is very similar to the more convenient synchronous programming model.

What makes all of this actually possible is C# iterators and an AsyncEnumerator class that I've developed to drive these iterators. Hopefully I've pumped this up so much that you're dying to see my AsyncEnumerator class and how it works. Unfortunately, I've run out of space in this month's column, so you'll have to wait until my next column where I'll explain my AsyncEnumerator class in detail.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), an architecture review, consulting, and training firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


.NET Matters: Debugging finalizers

.NET Matters
Debugging finalizers
Stephen Toub

Q I have a bunch of custom types that, for one reason or another, need to implement IDisposable. I want to make sure that the other developers on my team using this class always dispose of my types correctly. What can I do to warn one of my teammates if he forgets to call Dispose?

A For starters, Visual Studio® 2005 and FxCop can help with this if you perform static analysis on your code. Rule CA2000, described at msdn2.microsoft.com/ms182289, is "Dispose objects before losing scope," which checks to see if any local IDisposable objects are created and then not disposed of before all references to the object are out of scope. Though helpful, it's not a perfect solution, as there's only so much that can be detected through static analysis. What you really need is a way to warn developers using your types when those types are garbage collected without having been disposed, and to do that you can take advantage of finalizers.

In general, you should avoid implementing finalizers on your types unless you absolutely have to. And with SafeHandles in the Microsoft® .NET Framework 2.0, there are very few reasons you absolutely have to. There are, however, many scenarios in which you should implement IDisposable, including any scenario where your type owns a managed resource that itself implements IDisposable; in such a scenario, your type should provide a Dispose method that in turn calls Dispose on the contained resource, as shown in Figure 1. (For a much more in-depth look at implementing IDisposable, see Shawn Farkas's CLR Inside Out column in the July 2007 issue of MSDN® Magazine at msdn.microsoft.com/msdnmag/issues/07/07/CLRInsideOut.)

Figure 1 Disposing Contained Managed Resources (Close Figure 1)
public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass()
    {
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this); // if DisposableClass isn't sealed
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            _data.Dispose();
        }
    }

    ...
}
Figure 1 Disposing Contained Managed Resources (Close Figure 1)
public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass()
    {
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this); // if DisposableClass isn't sealed
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            _data.Dispose();
        }
    }

    ...
}

In Figure 1, DisposableClass doesn't directly use any native resources, and thus it doesn't need to provide a finalizer (its Dispose method does still call GC.SuppressFinalize in case a derived type implements a finalizer). However, by adding a finalizer for debugging purposes, you can introduce a way to find out when a class was not properly disposed, as shown in Figure 2. As long as all instances of the type are disposed of properly, the finalizer will never be called; if, however, any instance is not disposed of, the finalizer thread will call its Finalize method (~DisposableClass) after that instance is found to no longer be referenced, which will in turn cause an assertion in the debugger.

Figure 2 Adding a Finalizer for Debugging (Close Figure 2)
public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass()
    {
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            data.Dispose();
        }
    }

#if DEBUG
    ~DisposableClass()
    {
        Debug.Fail("You forgot to Dispose this instance.");
    }
#endif

    ...
}
Figure 2 Adding a Finalizer for Debugging (Close Figure 2)
public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass()
    {
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            data.Dispose();
        }
    }

#if DEBUG
    ~DisposableClass()
    {
        Debug.Fail("You forgot to Dispose this instance.");
    }
#endif

    ...
}

Unfortunately, while the developer using your type will now know that they missed disposing an instance, and they'll know which instance, they may not know where that instance came from. If your types are only being instantiated in one place, that may be sufficient. But if your types are being instantiated from all over the code base, the developers will have no way of knowing where this particular instance came from, making it more difficult for them to figure out how this instance slipped through.

To address that, you can collect additional information into your class at construction time. For example, you could add the following members to keep track of the stack trace, thread, and time at and from which this instance was created:

#if DEBUG
    private StackTrace _stack = new StackTrace(true);
    private int _threadId = Thread.CurrentThread.ManagedThreadId;
    private DateTime _time = DateTime.UtcNow;
    ...
#endif

You could also modify Debug.Fail to include information from these members when the finalizer is executed, providing the user a better chance of finding exactly where this instance came from and tracking down why it wasn't properly disposed. Note that the pragma directives in the previous code snippet (#if/#endif) are used to ensure that all of this code is only compiled into DEBUG builds, rather than RELEASE builds. Retrieving stack traces is a relatively expensive operation, and it's something you should avoid whenever possible if performance is a concern.

Additionally, you'll notice that in Figure 2 I've surrounded the finalizer in the same fashion. This is mostly a matter of style. Finalizers add some overhead to the system, even if they're suppressed, so there's no point in having one on our class if it's not necessary for correct functionality. (When allocated, finalizable objects are added to a finalization list. When these instances are no longer reachable and the GC runs, they're moved to the "FReachable" queue, which is processed by the finalizer thread. Suppressing finalization with GC.SuppressFinalize sets a "do not run my finalizer" flag in the object's header, such that the object will not get moved to the FReachable queue by the GC. As a result, while minimal, there is still overhead to giving an object a finalizer even if the finalizer does nothing or is suppressed.)

This works well for an individual type, but if you're creating lots of types that need similar behavior, most coding practices dictate that you factor out this code into a reusable class. This is actually a deceptively tricky problem, and I'll spend the rest of this answer walking through several different implementations, providing the pros and cons of each. You can pick and choose from each implementation based on your needs.

One thing that's common to all implementations is a need to store various amounts of information at construction time. For this purpose, I've chosen to create a custom exception class. This exception class, when constructed, retrieves data similar to that mentioned previously. It can then later be thrown when the finalizer is invoked, or its ToString method can be used as the argument to a call to Debug.Fail or something similar. (Note that with the .NET Framework 2.0 and later, throwing an exception from a finalizer thread will, by default, tear down the application. This is different than in previous versions, where exceptions from the finalizer thread would be silently eaten by the runtime. I've chosen not to throw the exception, but if you believe that not disposing an object is a serious enough error to warrant tearing down the process, feel free to uncomment the relevant line.) My InvokedFinalizerException is shown in Figure 3.

Figure 3 InvokedFinalizerException (Close Figure 3)
[Serializable]
public class InvokedFinalizerException : Exception
{
    public readonly DateTime InstantiationTime = DateTime.UtcNow;
    public readonly int InstantiationThreadId =
        Thread.CurrentThread.ManagedThreadId;
    public readonly StackTrace InstantiationStackTrace =
        new StackTrace(true);

    public InvokedFinalizerException() :
        base("A finalizer was invoked.") { }

    public InvokedFinalizerException(string message) : base(message) { }

    public InvokedFinalizerException(string message, Exception inner) :
        base(message, inner) { }

    protected InvokedFinalizerException(
        SerializationInfo info, StreamingContext context) :
        base(info, context) { }

    public override string ToString()
    {
        return
            "Time: " + InstantiationTime + Environment.NewLine +
            "Thread: " + InstantiationThreadId + Environment.NewLine +
            "Stack: " + InstantiationStackTrace;
    }
}
Figure 3 InvokedFinalizerException (Close Figure 3)
[Serializable]
public class InvokedFinalizerException : Exception
{
    public readonly DateTime InstantiationTime = DateTime.UtcNow;
    public readonly int InstantiationThreadId =
        Thread.CurrentThread.ManagedThreadId;
    public readonly StackTrace InstantiationStackTrace =
        new StackTrace(true);

    public InvokedFinalizerException() :
        base("A finalizer was invoked.") { }

    public InvokedFinalizerException(string message) : base(message) { }

    public InvokedFinalizerException(string message, Exception inner) :
        base(message, inner) { }

    protected InvokedFinalizerException(
        SerializationInfo info, StreamingContext context) :
        base(info, context) { }

    public override string ToString()
    {
        return
            "Time: " + InstantiationTime + Environment.NewLine +
            "Thread: " + InstantiationThreadId + Environment.NewLine +
            "Stack: " + InstantiationStackTrace;
    }
}

With that in place, the first technique involves moving all of the tracking code into a separate constructible class, FinalizationDebugger, as shown in Figure 4. An instance of this class is stored as a member of your disposable class and is constructed when your class is constructed. When your class is disposed, this separate FinalizationDebugger class is also disposed. If your class isn't disposed, when it's no longer referenced FinalizationDebugger will eventually be collected, and its finalizer will provide all of the relevant information we need to figure out what instance wasn't properly disposed of, where it came from, and so forth. Note that FinalizationDebugger holds a reference to your object, but since your object contains the only reference to this FinalizationDebugger instance, its reference to your object will not prevent your object from being collected (if the .NET Framework garbage collector instead worked on a reference counting scheme rather than the mark-and-sweep scheme it does use, this cycle could be a problem). Of course, as you can see in Figure 4, using this in your disposable class still requires adding a bit of goo. In fact, this isn't much better than our original version we were trying to refactor into a cleaner solution.

Figure 4 Member Approach (Close Figure 4)

Refactored into Separate Class

public sealed class FinalizationDebugger : IDisposable
{
    private InvokedFinalizerException _exc;
    private object _obj;

    public FinalizationDebugger(object obj)
    {
        _exc = new InvokedFinalizerException();
        _obj = obj;
    }

    public void Dispose()
    {
        GC.SuppressFinalize(this);
    }

    ~FinalizationDebugger()
    {
        Debug.Fail(_exc.ToString());
        // throw _exc;
    }
}

Using FinalizationDebugger
public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;
#if DEBUG
    private FinalizationDebugger _fd;
#endif

    public DisposableClass()
    {
#if DEBUG
       _fd = new FinalizationDebugger(this);
#endif
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            _data.Dispose();
#if DEBUG
            _fd.Dispose();
#endif
        }
    }
}

The next approach takes this one step further by moving all of this code into a base class that your disposable class can derive from. This approach is shown in Figure 5. Notice how clean this makes your disposable class, as the only changes required are adding FinalizationDebuggerBase as the base class for your type, deleting your parameterless Dispose method (since it's already implemented by the base class), and changing your Dispose(bool) method to override the base implementation.

Figure 5 Base Class Approach (Close Figure 5)

Refactored into Base Class

public class FinalizationDebuggerBase : IDisposable
{
#if DEBUG
    private FinalizationDebugger _fd;
#endif

    public FinalizationDebuggerBase()
    {
#if DEBUG
        _fd = new FinalizationDebugger(this);
#endif
    }

    public void Dispose() { Dispose(true); }

    protected virtual void Dispose(bool disposing)
    {
#if DEBUG
        if (disposing) _fd.Dispose();
#endif
    }
}

Using FinalizationDebuggerBase

public class DisposableClass : FinalizationDebuggerBase
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass() {}

    protected override void Dispose(bool disposing)
    {
        try
        {
            if (disposing) _data.Dispose();
        }
        finally { base.Dispose(disposing); }
    }
}

Unfortunately, there are several problems with this approach. First, the #if/#endif pragma directives are evaluated at compile time when FinalizationDebuggerBase is compiled. If FinalizationDebuggerBase is part of the same project as your DisposableClass, it will always pick up the same DEBUG/RELEASE compilation flags as your class and will be kept in sync, which is almost certainly what you want to happen. But if FinalizationDebuggerBase is in another assembly (for example, if it's compiled into an assembly that's shared by a bunch of projects at your company), the assembly you reference will have already been compiled, which means it won't respect your DEBUG/RELEASE choice for your project. As a result, you could compile your project in RELEASE mode and expect none of this finalization code to be included. But if the FinalizationDebuggerBase class were compiled in DEBUG mode, your expectations would not be met. The second problem, which should be more obvious, is that .NET does not support multiple class inheritance, meaning that a class can only have one base class (though it can implement any number of interfaces). This means if your class already has a base class, you can't use this approach (unless you can somehow finagle this base class into the hierarchy somewhere).

A third option involves factoring everything out into a static class that your instances call into. My implementation of this is shown in Figure 6. In this approach, FinalizationDebugger is a static class that exposes three static methods: Constructor, Dispose, and Finalizer. The idea is that you call these methods from the appropriate place in your class, where the appropriate place should be obvious from the names of these methods (see Figure 6 for an example). This is minimally invasive into your code, as it typically involves adding only three lines of code (though if your type doesn't already have a finalizer, you'd need to add those few additional lines). All of these methods are marked with a ConditionalAttribute such that they'll only be called by your class when you compile your class in DEBUG mode.

Figure 6 Static Class Approach (Close Figure 6)

Refactored into a Static Class

public static class FinalizationDebugger<T> where T : class
{
    private static Dictionary<object, InvokedFinalizerException> _db =
        new Dictionary<object, InvokedFinalizerException>();

    [Conditional("DEBUG")]
    public static void Constructor(T obj)
    {
        if (obj == null) throw new ArgumentNullException("obj");
        ObjectEqualityWeakReference weakRef =
            new ObjectEqualityWeakReference(obj);
        InvokedFinalizerException exc = new InvokedFinalizerException();
        lock (_db) _db.Add(weakRef, exc);
    }

    [Conditional("DEBUG")]
    public static void Dispose(T obj)
    {
        if (obj == null) throw new ArgumentNullException("obj");
        lock (_db) _db.Remove(obj);
    }

    [Conditional("DEBUG")]
    public static void Finalizer(T obj)
    {
        if (obj == null) throw new ArgumentNullException("obj");
        InvokedFinalizerException exc;
        lock (_db)
        {
            if (_db.TryGetValue(obj, out exc)) _db.Remove(obj);
        }
        if (exc != null)
        {
            Debug.Fail(exc.ToString());
            // throw exc;
        }
    }
    ... // private class ObjectEqualityWeakReference
}

Using FinalizationDebugger<T>

public class DisposableClass : IDisposable
{
    private SomeOtherDisposableClass _data = ...;

    public DisposableClass()
    {
        ...
        FinalizationDebugger<DisposableClass>.Constructor(this);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing)
        {
            FinalizationDebugger<DisposableClass>.Dispose(this);
            _data.Dispose();
            ...
        }
    }

    ~DisposableClass()
    {
        FinalizationDebugger<DisposableClass>.Finalizer(this);
        ...
    }
}

The implementation of this class deserves a bit of explanation. FinalizationDebugger contains a static Dictionary that maps objects to our custom exception class that contains all of the information about when and how that associated object was constructed. When an object is constructed and calls the Constructor method, passing itself as an argument, the object is inserted into this table along with an instance of the exception. When the Dispose method is called, the object and associated exception are removed from the table. When the Finalizer method is called, the table is queried to see if the object exists; if it does, the associated exception is retrieved and is used to assert. (It can also be thrown if desired.) There are, of course, a few intricacies to this implementation.

The first is that, as these are static methods and could conceivably be called from multiple threads concurrently, we need to protect access to shared data. To do so, I use a monitor to shield all accesses to the underlying Dictionary.

The second issue is more interesting. The Dictionary is a static member of the FinalizationDebugger class, which makes it a GC root. As such, any objects stored in this Dictionary will never be free for collection by the garbage collector, because they'll always be reachable. If I were to simply store your disposable class instances into this Dictionary, they would never be available for garbage collection, and thus they'd never be finalized, defeating this whole system (not to mention causing a huge memory leak). To solve this, we need weak references.

The System.WeakReference class takes advantage of functionality provided by the garbage collector and System.GCHandle. When you instantiate a WeakReference for an object, internally WeakReference allocates a weak GCHandle (either GCHandleType.Weak or GCHandleType.WeakTrackResurrection) for the object. Rather than storing a reference to your object, it simply stores this GCHandle. A weak GCHandle is used to track an object but still allow it to be collected (when an object is collected, the contents of the GCHandle are zeroed). Thus, WeakReference allows you access to the underlying object, but it doesn't maintain a strong reference to it, such that if the garbage collector runs and the object would otherwise be collectable, it'll be collected and WeakReference will return null on any future attempts to access the object. As such, WeakReferences solve our static Dictionary problem. By storing WeakReferences instead of the actual object, the objects will still be collectable.

That, however, leads to another problem. Dictionary uses an object's hash code as well as its Equals method to determine where in the Dictionary to store the object and whether the object already exists in the Dictionary (for lookups and the like). The WeakReference reference type, however, doesn't override either GetHashCode or Equals; its equality semantics default to that of System.Object, and thus checks to see whether the two WeakReferences instances being compared are the exact same instance. As a result, code such as the following will print out "False" twice:

object a = new object();
Dictionary<object, bool> dict = new Dictionary<object, bool>();
dict.Add(new WeakReference(a), true);
Console.WriteLine(dict.ContainsKey(a));
Console.WriteLine(dict.ContainsKey(new WeakReference(a)));

To work around this, there are two solutions we can attempt. The first would be to modify the FinalizationDebugger.Constructor method to return the created WeakReference. Your DisposableClass could then hold onto this WeakReference and provide it (rather than itself) in the future calls to FinalizationDebugger.Dispose and FinalizationDebugger.Finalizer:

private WeakReference _weakRef;
...
_weakRef = FinalizationDebugger.Constructor(this);
...
FinalizationDebugger.Finalizer(_weakRef);

This introduces another problem, however. The order in which objects are finalized is undefined, and, as a result, it's a bad idea to reference one finalizable instance (that could have already been finalized) in the finalizer of another. WeakReference itself implements a finalizer, and thus the previous code snippet results in attempting to use a WeakReference that may have already been finalized from the finalizer of your disposable type.

The second solution, and the one I've chosen to implement, solves the problem at its core: implementing GetHashCode and Equals on WeakReference. To do so, I've created the class shown in Figure 7, ObjectEqualityWeakReference, which overrides WeakReference's Equals and GetHashCode methods to provide the semantics I need. When ObjectEqualityWeakReference is constructed, it caches into a member variable the hash code of the supplied object, and it is this cached value that's returned from the overridden GetHashCode; that way, the hash code is based on the underlying object. And even if the object is collected, the ObjectEqualityWeakReference will continue to return the same hash code value. An object's hash code should never change, and Dictionary uses the hash code as the first step to finding an object in its table; if the object's hash code changes after the object has been added, there's a good chance Dictionary won't be able to find it.

Figure 7 ObjectEqualityWeakReference (Close Figure 7)
private class ObjectEqualityWeakReference : WeakReference
{
    private int _hashCode;

    public ObjectEqualityWeakReference(object obj) :
        base(obj, true)
    {
        _hashCode = RuntimeHelpers.GetHashCode(obj);
    }

    public override bool Equals(object obj)
    {
        WeakReference other = obj as WeakReference;

        if (other == null) return ReferenceEquals(obj, this.Target);

        return ReferenceEquals(other, this) ||
               ReferenceEquals(other.Target, this.Target);
    }

    public override int GetHashCode() { return _hashCode; }
}
Figure 7 ObjectEqualityWeakReference (Close Figure 7)
private class ObjectEqualityWeakReference : WeakReference
{
    private int _hashCode;

    public ObjectEqualityWeakReference(object obj) :
        base(obj, true)
    {
        _hashCode = RuntimeHelpers.GetHashCode(obj);
    }

    public override bool Equals(object obj)
    {
        WeakReference other = obj as WeakReference;

        if (other == null) return ReferenceEquals(obj, this.Target);

        return ReferenceEquals(other, this) ||
               ReferenceEquals(other.Target, this.Target);
    }

    public override int GetHashCode() { return _hashCode; }
}

ObjectEqualityWeakReference also overrides the Equals method. I wanted Equals to work when comparing an instance of ObjectEqualityWeakReference to another WeakReference referencing the same underlying object or when comparing it directly against that object. As such, Equals first checks to see if the object being compared is another WeakReference. If it is, Equals returns whether the WeakReference instances are the same or whether the underlying objects are the same. If the object being compared is not a WeakReference, Equals returns whether it matches the underlying object. (Wanting to support adding ObjectEqualityWeakReference instances but querying by the underlying object type is the reason that the Dictionary's TKey type parameter is bound to Object rather than to ObjectEqualityWeakReference.)

There's one more important thing to note about ObjectEqualityWeakReference, and that's its constructor:

public ObjectEqualityWeakReference(object obj) : base(obj, true)

This constructor delegates to the constructor of WeakReference that accepts two parameters, not one. The second parameter is a Boolean that indicates to WeakReference when to stop tracking the object. If the value is false, the object is only tracked until finalization, and if the value is true, the object is tracked until after finalization. Because we need the object to survive through the call to FinalizationDebugger.Finalize, we set this to true. Note that this Boolean parameter simply dictates what kind of GCHandle is created. If the value is false, a GCHandle.Weak is created, and if it's true, a GCHandle.WeakTrackResurrection is created. The naming comes from the concept of resurrection, whereby an object can be saved during finalization. If an object's finalizer creates a new rooted reference to the object (such as by storing the object into a static field somewhere), the object will now again be reachable and thus won't be collected. In order for that to work with a WeakReference, the GCHandle must not allow the object to be collected until after the finalization phase, and hence naming the previously described behavior "WeakTrackResurrection."

That completes the static class solution, which works well. It does, however, have drawbacks just like the other approaches. For one, due to the static Dictionary that needs to be protected, this approach requires locks, which could cause contention if lots of objects are being constructed and disposed. This is mitigated to some extent in a few ways. First, the work performed in the locked regions is kept to a minimum—I'm simply wrapping a Dictionary.Add call in Constructor, a Dictionary.Remove call in Dispose, and Dictionary.TryGetValue and Dictionary.Remove calls in Finalizer. Second, I've made FinalizationDebugger a generic class based on the type of the objects being tracked. This, in effect, creates a whole new FinalizationDebugger type per type you're working with (with separate locks and dictionaries), and as such only those calls based on the same type will cause contention.

Another drawback to this approach is that locking in a finalizer is not the greatest thing to do. If for some reason the finalizer isn't able to obtain the lock within a reasonable amount of time, the CLR may simply abort the finalizer, but that's very unlikely given the current code.

The biggest drawback I see with this approach is that it requires your disposable class to implement a finalizer so that it can call the FinalizationDebugger.Finalizer method. If the Finalizer method is never called, the check will never be performed to see whether the object wasn't properly disposed. This isn't a big deal if your class already implements a finalizer, but if it doesn't, you'll need to add one for this purpose.

There are other variants on all of these approaches, and I'm sure there are other approaches I haven't considered. If you have a new approach that you think solves all of the various problems each of these approaches has, I'd be interested in hearing about it. In the meantime, any of these should help you to enable developers using your disposable types to track down cases where they are not properly disposing your objects.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


{ End Bracket }: WPF is for developers, too

{ End Bracket }
WPF is for developers, too
Kevin Moore

It's been a lot of fun seeing the excitement building around Windows® Presentation Foundation (WPF) in its first months in the wild. The introduction of Expression Blend™ has solidified WPF as the first user experience platform from Microsoft built with designers in mind. This designer empowerment has been made possible by XAML, an XML-based model for describing objects. In the case of WPF, XAML describes controls, styles, and animations. Being XML-based enables XAML to be much more toolable than the C# or Visual Basic® code that is generated for Windows Forms applications, which means designers can focus on the appearance and interaction of an application instead of the programming details.

This XAML-based future is not without its detractors, though. I've chatted with more than one frustrated developer who laments, "I just want to write code." That reaction probably results from a combination of fear of the unknown and lack of knowledge. As someone who joined WPF halfway through development, I can attest to a bit of XAML skepticism. "What? Another way to instantiate objects?" Believe me, XAML is to be embraced, not feared. When you want to understand the layout of an application, it's a lot easier to understand a page of nested elements and attributes than a page of squiggly braces and semicolons. You also gain portability as XAML is designed to be usable across tools.

So where does the developer plug in? Are fans of Visual Basic and C# left to write event handlers for buttons? Not at all. At the Mix 2007 conference in Las Vegas, I outlined three roles that let developers deliver real, non-trivial benefits to WPF applications while providing a great "better-together" experience with XAML content. These roles feature the developer as plumber, as widget author, and as groundbreaker.

The developer as plumber is the idea that many developers are not interested in UIs and prefer instead to enable designers by building powerful data layers. Suppose you want to build an application to track a set of stocks. By defining each stock object with public properties, it's easy to define data templates to visualize each stock. If the objects implement INotifyPropertyChange, the visualization will transparently update as the data changes. You can leverage WPF command infrastructure, via the ICommand interface, to allow declarative access to business object behaviors. For instance, instead of just exposing a Refresh method, expose a RefreshCommand. This can be connected to a button or menu without writing event handlers. Commanding will even disable the associated UI when the action is not available, which gives someone using design tools the maximum flexibility in the creative process while isolating business logic.

A savvy designer working with Expression Blend and a well-defined data layer can go a long way. Most of the in-box controls can be tweaked with styles and controls templates. Binding and data templates enable rich visualization of business objects. In the end, though, a designer is limited to the controls and elements—the "widgets"—that WPF ships with. But developers can drastically improve the effectiveness of a XAML-defined application by defining new, custom widgets to target a given scenario. Imagine a design that calls for an arbitrary number of elements to be arranged in a circle. The naive way to approach this would be to build the elements using drawing APIs. Though this would be perfectly fine for a constrained scenario, it is limited. If the appearance of a child element needs to change, the code needs to change. A better approach would be to subclass Panel. Now you have an element that can be used on arbitrary visual elements, in a design tool, and with an ItemsControl to display arbitrary data items. This is just one example. There are many ways to address design challenges in WPF. By understanding how various components work together, a developer can design widgets with the maximum flexibility.

If "plumber" is coding to stay out of the way and "widget author" is coding to play well with others, "groundbreaker" is stealing the show. At its core, WPF has a powerful 2D/3D graphics engine. While important pieces of this engine have been exposed via controls and other framework features for XAML-based tools, much more capability lays waiting for a clever coder to unleash. Whether it's leveraging interoperability between 2D and 3D to build novel data visualizations or rolling custom animations to provide organic transitions, the developer—compiler in hand—has the most power and flexibility in the WPF world. Combined with a strong vision and a set of best practices, there is no limit to what designers and developers can accomplish in WPF.

Kevin Moore is a Program Manager on the Windows Presentation Foundation team at Microsoft, where he works on data and controls. Visit his blog at work.j832.com.