November 2008 November 2008
Security Quiz: Test Your Security IQ
Our security experts present 10 vulnerable pieces of code. Your mission is to find the holes (a.k.a. bad security practices) in the code. Michael Howard and Bryan Sullivan
Agile SDL: Streamline Security Practices For Agile Development
Bryan Sullivan discusses the new SDL for Web applications and Agile projects with more compressed release cycles. Bryan Sullivan
Access Control: Understanding Windows File And Registry Permissions
Understanding the ACLs that govern permissions and rights before an operation is allowed to proceed is critical to enhancing security. John R. Michener
Utility Spotlight: 12 Steps To Faster Web Pages With Visual Round Trip Analyzer
Download Visual Round-trip Analyzer (VRTA) to uncover the root of your Web page loading problems and identify these 12 common ailments. Jim Pierson
RIA: Light Up SharePoint With Silverlight 2 Web Parts
Find out how to integrate SharePoint and Silverlight by creating a Silverlight media player and deploying it as a SharePoint Web Part. Steve Fox and Paul Stubbs
Columns
Editor's Note: Can I See Some Identification?: Security measures are highly context driven and change with circumstances. This month Howard Dierking spins a few security yarns to illustrate.Howard Dierking | Toolbox: Online Scheduling, Jeff Smith’s SQL Blog, Easy Regular Expressions, And More: This month Scott looks at improving development skills, writing regular expressions, a web scheduling control and a SQL tips blog.Scott Mitchell | CLR Inside Out: International Applications: The CLR team takes a look inside the System.Globalization namespace to explain how to handle data formats for proper localization and globalization.Melitta Andersen
Data Points: Cloud Gazing From Silverlight 2: John Papa tackles questions about calling services from Silverlight 2 applications.John Papa | Cutting Edge: Browser Interoperability In Silverlight 2: This month Dino Esposito explains how the browser interoperability layer in Silverlight addresses a number of your Silverlight / Web page interaction needs.Dino Esposito | Security Briefs: Threat Models Improve Your Security Process: Using threat models to drive your security engineering process helps prioritize the code review, fuzz testing, and attack surface analysis tasks.Michael Howard
Test Run: Group Determination In Software Testing: There is a large body of research on group techniques to determine the best alternative from a set of options. Dr. James McCaffrey outlines five of them.Dr. James McCaffrey | Wicked Code: Silverlight 2 Transforms And Clipping Regions: Silverlight is powerful enough to let you easily build an image magnification feature for you web site with very little code, most of which is XAML. Find out how.Jeff Prosise | Foundations: Unit Testing Workflows And Activities: Matt Milner takes a look at some of the challenges and techniques related to testing Windows Workflow Foundation activities, workflows, and associated components.Matt Milner
Team System: Team Build 2008: Brian Randell introduces you to Visual Studio 2008 Team Foundation Server Build (Team Build). Then he creates and runs a team build.Brian A. Randell | { End Bracket }: Engineers Who Write: Writing is good exercise for developers and software engineers. Richard Ward explains how writing can help you focus your thoughts and communicate more precisely.Richard Ward


Security Quiz
Test Your Security IQ
Michael Howard and Bryan Sullivan
Contents
Bug #1 (C or C++)
Bug #2 (C or C++)
Bug #3 (could be any language—example is in C#)
Bug #4
Bug #5
Bug #6 (C#)
Bug #7 (C#)
Bug #8 (C#)
Bug #9 (C#)
Bug #10 (Silverlight CLR C#)

Both of us enjoy reviewing code for security bugs. We would even go as far as to say we're pretty good at it. We don't pretend we're the best, but we can generally find plenty of bugs pretty quickly. Can you?
Would you know a security bug if you saw one? Find out by taking this quiz. Each code sample has at least one security vulnerability. Try to spot the bugs and see how you rate. Following the code is a summary of any vulnerabilities, some commentary, and, where appropriate, how the Security Development Lifecycle (SDL) can help find these bugs. Thanks to Peter Torr and Eric Lippert for providing input and code samples.
Bug #1 (C or C++)
void func(char *s1, char *s2) {
  char d[32];
  strncpy(d,s1,sizeof d - 1);
  strncat(d,s2,sizeof d - 1);
  ...
}
Answer We thought we'd start off pretty easy on you with a good ol' buffer overrun. To many people, the code is fine and secure because the code uses the bounded strncpy and strncat functions. However, these functions are only secure if the buffer sizes are correct, and in this example the buffer sizes are wrong. Dead wrong.
Technically, the first call is secure, but the second call is wrong. The last argument of the strncpy and strncat functions is the amount of space left in the buffer, and you just exhausted some or all of that space with the call to strncpy. Buffer overflow. Michael blogged about this exact bug type in 2004.
In Visual C++ 2005 and later, warning C4996 tells you to replace the bad function call with a safer call, and the /analyze option would issue a C6053 warning indicating that strncat might not zero terminate the string.
To be perfectly honest, strncpy and strncat (and their "n" cousins) are worse than strcpy and strcat (and their ilk) for a couple of reasons. First, the return value is just silly—it's a pointer to a buffer, a buffer that might be valid or might not. You have no way of knowing! Second, it's really tough to get the destination buffer size right. If you spotted the bug, give yourself one point.

Bug #2 (C or C++)
int main(int argc, char* argv[]) {
  char d[32];
  if (argc==2)
    strcpy(d,argv[1]);

  ...
  return 0;
}
Answer This is an evil bug. We have seen this bug used so many times as an example of a buffer overrun, and most of the time it's impossible to determine if there is a security bug in the code or not. It all depends on how the code is used.
If this is a standard Win32 EXE, then there is no security bug here because the worst you can do is attack yourself and run code as yourself, which isn't a security bug.
Now, if this code were located in ServiceMain of a Windows service running as SYSTEM, for example, or the main function of a Linux application setuid root, then the code becomes a bona fide security bug.
Let's say the code is a Linux application marked setuid root. When the application is started by a normal user, the application actually runs as root, which means this is a local privilege elevation vulnerability.
As in the code example given in Bug #1, C4996 warnings are issued for the call to strcpy, and /analyze would issue a C6204 indicating a potential buffer overrun. If you answered, "Huh! I need much more context," then give yourself two points; otherwise, no points for you.

Bug #3 (could be any language—example is in C#)
byte[] GetKey(UInt32 keySize) {
  byte[] key = null;

  try {
    key = new byte[keySize];
    RNGCryptoServiceProvider.Create().GetBytes(key);
  }
  catch (Exception e) {
    Random r = new Random();
    r.NextBytes(key);
  }

  return key;
}
Answer There are two bugs in this lousy key-generation code. The first is pretty obvious: if the call to the cryptographically sound random number generator fails, the code catches the exception and then calls a truly lousy and predictable random generator. If you spotted this, give yourself a point. It's an SDL requirement to use cryptographically random numbers to generate keys.
But there's another bug: the code catches all exceptions. Other than in rare instances, catching all exceptions—whether they are C++ exceptions, Microsoft .NET Framework exceptions or structured exception handling on Windows—hides real errors. So don't do it.
A structured exception handler in C or C++ that catches all exceptions (including access-protection faults such as buffer overruns) will yield a C6320 warning when compiled with /analyze. It was this kind of design that let attackers reattempt their attacks against the animated cursor bug MS07-017. If you spotted the exception-handling bug, give yourself one more point.

Bug #4
void func(const char *s) {
  if (!s) return;
  char t[3];
  memcpy(t,s,3);
  t[3] = 0;
  ...
}
Answer We found a bug like this one a few years ago in Windows Vista while it was still in the process of being developed. But is this a security bug? Obviously it's a coding bug because the code writes to the fourth array element, and the array is only three elements long. Remember, arrays start at zero, not one. I would contend that this is not a security bug because the attacker has no control whatsoever.
If the bug looked like this where the attacker controls i, however, then that would mean the attacker could write a zero anywhere in memory. And that's a card-carrying security bug:
void func(const char *s, int i) {
  if (!s) return;
  char t[3];
  memcpy(t,s,3);
  t[i] = 0;
  ...
}
This code yields a C6201 "out of valid index range" warning when compiled with /analyze. If you said, "Not a security bug," give yourself one point.

Bug #5
public class Barrel {
  // By default, a barrel contains one rhesus monkey.
  private static Monkey[] defaultMonkeys =
    new[] { new RhesusMonkey() };
  // backing store for property.
  private IEnumerable<Monkey> monkeys = null;

  public IEnumerable<Monkey> Monkeys {
    get {
      if (monkeys == null) {
        if (MonkeysReady())
          monkeys = PopulateMonkeys();
        else
          monkeys = defaultMonkeys;
      }
      return monkeys;
    }
  }
}
Answer This is a hard one. The author of this class thinks that they are being both safe and efficient. The backing store is private, the property is read-only, and the property type is IEnumerable<T>, so the caller cannot do anything but read the state of the Barrel.
The author has forgotten that a hostile caller can try to cast the return value of the property to Monkey[]. If there are two Barrels and each one has the default Monkey list, then a hostile caller that has one of them can replace the RhesusMonkey in the static default list with any other Monkey, or null, thereby effectively changing the state of the other Barrel.
The solution here is to cache a ReadOnlyCollection<T> or some other truly read-only storage that protects the underlying array from mutation by a hostile or buggy caller. If you got this, give yourself two points.

Bug #6 (C#)
protected void Page_Load(object sender, EventArgs e) {
  string lastLogin = Request["LastLogin"];
  if (String.IsNullOrEmpty(lastLogin)) {
    HttpCookie lastLoginCookie = new HttpCookie("LastLogin",
      DateTime.Now.ToShortDateString());
    lastLoginCookie.Expires = DateTime.Now.AddYears(1);
    Response.Cookies.Add(lastLoginCookie);
  }
  else {
    Response.Write("Welcome back! You last logged in on " + lastLogin);
    Response.Cookies["LastLogin"].Value =
      DateTime.Now.ToShortDateString();
  }
}
Answer This is a straightforward, cross-site scripting vulnerability, the most common vulnerability on the Web. Although the code seems to imply that the lastLogin value always comes from a cookie, in fact, the HttpRequest.Item property will prefer a value from the query string over a value from a cookie.
To put this another way, no matter what the value of the lastLogin cookie happens to be set to, if an attacker adds the name/value pair lastLogin=<script>alert('0wned!')</script> to the query string, the application will choose the malicious script input for the value of the lastLogin variable. If you answered XSS, give yourself one point.

Bug #7 (C#)
private decimal? lookupPrice(XmlDocument doc) {
  XmlNode node = doc.SelectSingleNode(
    @"//products/product[id/text()='" +
    Request["itemId"] + "']/price");
  if (node == null)
    return null;
  else
    return (Convert.ToDecimal(node.InnerText));
}
Answer If you said XPath injection, give yourself one point. XPath injection works on exactly the same principle as its much more (in)famous cousin SQL injection. By creating a query that combines XPath code with unvalidated and unescaped user input, this code is vulnerable to injection attacks. Any application that manipulates text that is then used to perform some form of operation is subject to injection vulnerabilities.

Bug #8 (C#)
public class CustomSessionIDManager : System.Web.Session    State.SessionIDManager
{
    private static object lockObject = new object();

    public override string CreateSessionID(HttpContext context)
    {
        lock (lockObject)
        {
            Int32? lastSessionId = (int?)context.Application                ["LastSessionId"];
            if (lastSessionId == null)
                lastSessionId = 1;
            else
                lastSessionId++;
            context.Application["LastSessionId"] = lastSessionId;
            return lastSessionId.ToString();
        }
    }
}
Answer There are two main problems here. While the code correctly applies a lock around the app logic to ensure that two threads don't create the same session ID at the same time, it's still not safe to deploy in a server farm. Application state as referenced by the HttpContext.Application object is not shared across servers. If this application was deployed in a server farm, it could lead to session ID collisions. If you caught this bug, give yourself one point.
Another serious problem is that the session IDs generated by this class are easily guessed sequential integers. If a user looks at his session token and notices that he has session ID 100, he could use a simple browser utility to change the session ID to 99 or 98 or any other lower value to hijack those users' sessions.
A much better option for the developer in this case would be to use a GUID or other large, random, string-combining letters and numbers. If you realized that sequential integers are poor choices for session ID tokens, you score a point.

Bug #9 (C#)
bool login(string username,
           string password,
           SqlConnection connection,
           out string errorMessage) {
  SqlCommand selectUserAndPassword = new SqlCommand(
    "SELECT Password FROM UserAccount WHERE Username = @username",
    connection);
  selectUserAndPassword.Parameters.Add(
    new SqlParameter("@username",  username));
  string validPassword =
    (string)selectUserAndPassword.ExecuteScalar();

  if (validPassword == null) {
    // the user doesn't exist in the database
    errorMessage = "Invalid user name";
    return false;
  }
  else if (validPassword != password) {
    // the given password doesn't match
    errorMessage = "Incorrect password";
    return false;
  }
  else {
    // success
    errorMessage = String.Empty;
    return true;
  }
}
Answer The biggest problem here is that the application is returning too much information to the user in the case of a failed login. While it's definitely helpful for a user to know whether he just mistyped his password or whether he completely forgot his user name, this information is also useful to an attacker attempting a brute force attack against the application. Although it sounds counterintuitive, in this situation it's better to be unhelpful. Failed logins should display messages such as "Invalid username or password," not "Invalid username" and "Invalid password."
If you caught this, you get a point. And give yourself a bonus point if you also remembered that the application shouldn't be storing passwords in plaintext in the database; instead, it should be storing and comparing salted hashes of the passwords.
So How Do You Rate?
Points | Comment
15+ | We're hiring.
11-14 | Not bad, not bad at all. Now apply your talents to all that code your buddies wrote.
7-10 | Hmmm. OK. You clearly have a lot to learn, but you're probably better than 50% of the developers out there writing Web apps today.
4-6 | You have a LOT to learn. Go to your favorite book store and buy all the books written by both authors of this article.
0-3 | Step away from the editor and compiler, and no one will get hurt.

Bug #10 (Silverlight CLR C#)
bool verifyCode(string discountCode) {
  // We store the hash of the secret code instead of
  // the plaintext of the secret code.
  // Hash the incoming value and compare it against
  // the stored hash.
  SHA1Managed hashFunction = new SHA1Managed();
  byte[] codeHash =
    hashFunction.ComputeHash(
      System.Text.Encoding.Unicode.GetBytes(discountCode));
  byte[] secretCode = new byte[] {
    116, 46, 130, 122, 36, 234, 158, 125, 163, 122,
    157, 186, 64, 142, 51, 153, 113, 79, 1, 42 };

  if (codeHash.Length != secretCode.Length) {
    // The hash lengths don't match, so the strings don't
    // match this should never happen, but we check anyway
    return false;
  }

  // perform an element-by-element comparison of the arrays
  for (int i = 0; i < codeHash.Length; i++) {
    if (codeHash[i] != secretCode[i])
      return false; // the hashes don't match
  }

  // all the elements match, so the strings match
  // the discount code is valid, inform the server
  WebServiceSoapClient client = new WebServiceSoapClient();
  client.ApplyDiscountCode();

  return true;
}
Answer The developer made a wise decision not to embed the secret code in plaintext in the code. If you only need to test whether a user knows a secret (such as a discount code or a password), it's always better to store a hash of that secret and compare hashes rather than storing the plaintext and comparing strings directly. Unfortunately, the developer chose to use the SHA-1 hash algorithm, which is showing serious signs of weakness and has been subsequently banned by the SDL. A much better choice would have been to use the SHA256Managed class that implements the SDL-approved and recommended SHA-256 hash algorithm. If you caught this, you get a point.
Even worse than choosing SHA-1 over SHA-256 is the fact that the developer neglected to salt the hash value. Unsalted hashes are much more vulnerable to cracking via precomputed hash tables (often called rainbow tables). It would probably take an attacker a short time to determine the original plaintext secret code from the unsalted hash value. (The authors will post congratulations in the SDL blog to acknowledge the first person who responds to us with the plaintext secret code.) Give yourself a point if you caught the unsalted hash.
However, the biggest problem with this code is the fact that it's being executed on the client machine at all! Remember we stated at the beginning that this is Silverlight CLR code that runs in the user's browser. Any code that runs on the client can be manipulated by an attacker. Period. Nothing prevents a determined user from attaching a debugger to the browser instance running the Silverlight code and stepping through the code as it executes.
He could set the codeHash variable to be equal to the secretCode hash so that the comparison logic would always succeed. Or he could skip over the validation logic completely and just jump the current instruction to the Web service call that applies the discount code. Easiest of all, he could avoid the debugger entirely and just call the Web service method ApplyDiscountCode directly!
It's important to realize that even though you may be using C# or Visual Basic to create Silverlight applications just like you would for ASP.NET Web Forms, Silverlight code runs on the client's machine, and Web Forms code runs on the server. Code that runs on the client can be viewed and changed by an attacker. Never embed secrets into client-side code or allow client-side code to make privileged decisions (such as whether a discount code is valid or whether a user is authorized to perform a certain action). If you caught this bug, give yourself a point.


Agile SDL
Streamline Security Practices For Agile Development
Bryan Sullivan

This article discusses:
  • Agile security
  • Threat modeling
  • Automating security processes
This article uses the following technologies:
SDL
Contents
A New Process: Combining the SDL and Agile
The Every-Sprint Requirements
The Onboarding Requirements
The Bucket Requirements
Constraints
Wrap-Up
In the September 2008 issue of MSDN Magazine, I wrote a column about the additions that Microsoft has made to the Security Development Lifecycle (SDL) process to address security vulnerabilities in online services. I talked about the importance of input validation and output encoding in order to prevent cross-site scripting attacks; about using parameterized stored procedures and restricting database permissions in order to defend against SQL injection attacks. I also discussed restricting the use of wildcards in cross-domain policy files so you can defend against request forgery attacks (see "SDL Embraces The Web").
All of these SDL additions are necessary to protect your Web apps, but they raise challenges for Web app development teams. In order to make the SDL more practical for Web app and online services teams, the process itself needs to change to better fit the development processes that those teams use. In other words, it's not just that the SDL needs new, Web-specific requirements; it's that those requirements need to be applied in a different manner as well.
The biggest difficulty in adapting the SDL to the needs of Web applications is simply one of time. The SDL was originally developed to improve the security of large, complex products like Windows, Microsoft Office, and SQL Server, and it has done so very successfully. Part of the reason for its success is its thoroughness: in its latest version, the SDL has more than 80 separate requirements and recommendations that product teams follow to improve their products' security and privacy. Figure 1 gives you an idea of the process involved in the SDL.
Figure 1 The Classic Security Development Lifecycle
For a product like SQL Server that has a two- to three-year development cycle, completing 80 requirements over the course of the release schedule is not an unfair burden. However, most Web applications don't have such long development cycles; they have two- to three-week development cycles. (In fact, some online services teams at Microsoft have development cycles as short as one week.) So while it may not be a problem for Office to complete 80 SDL requirements over three years, it would be completely unreasonable to expect an online service like Office Live to complete 80 requirements every three weeks. And even if they could complete this much security work, they wouldn't have much time left to implement any new features. Security is vitally important, but we can't focus on security to the complete exclusion of everything else.

A New Process: Combining the SDL and Agile
To address this issue, the SDL team has adapted the SDL to better suit the needs of teams with Rapid Application Development (RAD) or Agile development processes. We haven't yet determined an official name for this new SDL variant, but for the purposes of this article we'll refer to it as SDL/Agile, or SDL/A for short. Our intention is that any product team with an iterative, phaseless development methodology will be eligible to opt into the SDL/A process instead of the "classic" version of SDL. This does not limit SDL/A to only Web applications, since many box products also follow Agile development methodologies.
The first and biggest difference between classic SDL and SDL/A is that in SDL/A, every requirement does not need to be completed for every release (or every "sprint"). This may seem controversial. After all, we've always said that every requirement in the SDL is important and can't be skipped.
It is true that every SDL requirement is included because it has been proven to prevent a serious security or privacy vulnerability. The requirement to avoid certain banned APIs is included because it has been proven to prevent buffer overflows. The requirement to encode user input before displaying it on a Web page has been proven to prevent cross-site scripting attacks. However, as I discussed, there simply isn't enough time in a short release cycle to complete every SDL requirement and still do feature development. So, SDL/A defines three levels of requirement frequency (that is, how often the requirement must be completed), and each SDL requirement is placed into one of those three categories.

The Every-Sprint Requirements
The first frequency level for SDL/A is the "every-sprint" level. These requirements are considered non-negotiable and must be completed every sprint regardless of how short the sprint length is. These requirements place the highest burden on the development teams since they're being completed so frequently, so these requirements are chosen very carefully according to two factors.
The first factor is the importance of the requirement—in other words, how many vulnerabilities does this requirement prevent, and how critical are those vulnerabilities? Every requirement is important, but just as different vulnerabilities have different weights on the bug bar, the requirements that defend against those vulnerabilities have different weights as well.
For example, a temporary Denial of Service (DoS) attack is considered a much less severe attack than a remotely executed elevation of privilege. An SDL requirement that would prevent the elevation of privilege attack is much more likely to be included in the every-sprint frequency category. We also look at historic vulnerability data: if a requirement would have prevented one of the critical security vulnerabilities, such as those that were exploited by the Slammer or Blaster worms (if the requirement had existed in time to prevent introduction of the vulnerability), then that requirement is almost certain to be included in the every-sprint category.
One example of a requirement that falls into the every-sprint category is the requirement to use the ASP.NET-provided ValidateRequest cross-site scripting (XSS) defense. The ValidateRequest requirement meets both criteria for inclusion in every sprint: it helps products to defend against highly critical XSS attacks, and it's essentially free since ValidateRequest is enabled by default.
The second factor that contributes to a requirement being placed in the every-sprint category is the ease with which that requirement can be automated. Even if a requirement is unlikely to prevent another Code Red attack, if it can be automated into the development build process or code check-in policy, then it's likely to be required every sprint.
Another example of an every-sprint requirement is the requirement to threat model the application. Unlike enabling ValidateRequest, threat modeling can take a significant amount of time and effort. However, threat modeling is the cornerstone of the SDL and can never be omitted. Threat modeling is essential to uncovering design-level vulnerabilities that might require significant re-architecture in order to be mitigated if they're found later in the development lifecycle. Like heart surgery, threat modeling is expensive, but it's well worth the price!
In many cases, the cost of threat modeling is not even that high. One of the tenets of the Agile development process is to plan for no more work than is going to be completed during the current sprint. This goes for threat modeling too. There's no need to create a detailed threat model of the complete system during the very first sprint. Only the functionality being developed in the current sprint needs to be threat modeled for that sprint. In this way, the threat modeling requirement is self-regulating with regard to time spent.

The Onboarding Requirements
The second group of requirements are referred to as "onboarding" requirements. These are requirements that product teams have to complete once at the beginning of the project and then never need to address again. In the classic SDL, every requirement works this way. But products using classic SDL typically have well-defined development phases, including a well-defined end (release of the product). In contrast, Web applications often do not have well-defined endings. The team may release a new update every Friday afternoon and then start developing new features Monday morning, and this process can continue indefinitely.
An example of a one-time-only onboarding requirement is the requirement to configure the product's bug tracking system to track security-specific data. At Microsoft, all security defects found during development that are rated as Critical, Important, or Moderate in the product's bug bar must be fixed before release. But if you can't track which defects are security issues and which are functional issues, you can't reliably fix the security issues before release. Additionally, the security-specific defect fields, such as Security Defect Cause (buffer overflow, directory traversal, SQL injection, and so on) and Security Defect Effect (any of the STRIDE values), can be valuable to help developers fix the defect and to ensure that it doesn't happen again later. The team only needs to set up the tracking system this way once, no matter how long the project runs.
Another, slightly more involved onboarding requirement is the requirement to create a baseline threat model for the application. If the application is a new project, if no code has ever been written for it before, then this requirement is superfluous: the team only needs to threat model new features as they're designed on a sprint-by-sprint basis. However, a product team may be applying the SDL/A process for the first time to a previously existing project. In this case, a baseline threat model must be built to cover the preexisting functionality of the application.
Several other onboarding requirements, such as developing a security incident response plan, can be nontrivial as well. In order to prevent teams from spending a disproportionate amount of time on security, especially in their first stages with a new, unfamiliar SDL process, SDL/A provides for a three-month grace period for teams to complete the onboarding requirements.

The Bucket Requirements
All other SDL requirements that do not fall into either the every-sprint or onboarding requirement categories are placed into one of three requirement buckets: Security Verification, Design Review, and Response Plans. So, for example, the SDL requirement to fuzz input handling routines would be placed in the Security Verification bucket. Likewise, the requirement to define a disaster recovery plan would be placed in the Response Plans bucket.
Unlike in classic SDL, where all of these requirements must be completed before the product can release, in SDL/A only one requirement from each bucket must be completed during each sprint. This is the concession that SDL/A makes to the shorter release schedules of Agile development projects. As they say, you can't fit ten pounds of flour in a five-pound sack, and the bucket requirements are the five extra pounds we've taken out. Figure 2 shows the new flow with SDL/A. Note that the checked items will be performed during the corresponding stage.
Figure 2 SDL/A Lifecycle
However, even though product teams aren't required to complete every bucket requirement in every sprint, that doesn't mean they can omit them indefinitely. In fact, teams have to complete each of the bucket requirements at least once a year. Additionally, teams are prohibited from completing the same bucket requirement two sprints in a row. For example, if Project X's team chooses to complete an attack surface analysis to satisfy its Security Verification bucket requirement for their March sprint, they can't then complete another attack surface analysis in April. More specifically, they could complete another attack surface analysis in April if they really wanted to; nevertheless, it just would not satisfy their SDL commitment.
Other than those two constraints, teams are free to choose whichever bucket requirements they want to complete in any given sprint. SDL/A does not mandate any sort of ordering or round-robin requirement selection. This gives product teams maximum flexibility in choosing security activities that they find most useful. So if a team has historically found a significant number of defects by running Application Verifier but they've never found any by running a SOAP fuzzer, they can choose to run AppVerif as often as every other sprint but need only run the SOAP fuzzer once a year.

Constraints
While it's not officially prohibited, teams that follow a waterfall-style development process with well-defined and distinct design, implementation, verification and release phases are not a good fit for SDL/A because it is designed to work with iterative, phaseless development methodologies. If a product has well-defined phases, then classic SDL is a much better fit.

Wrap-Up
The concept of Agile security does not have to be a contradiction in terms. With SDL/A, the Microsoft SDL team has defined a set of process improvements that provide for an increased security focus while still respecting the need to release new code (including new functionality, not just new security improvements) on an ultra-short timeline. If you have experience, whether positive or negative, completing the classic SDL requirements in a short sprint, I'd be more than happy to hear your thoughts on SDL/Agile; you can reach me at blogs.msdn.com/sdl. Also, please see the official SDL site at www.microsoft.com/sdl.

Bryan Sullivan is a Security Program Manager for the Microsoft Security Development Lifecycle team, where he specializes in Web application security issues. His first book, Ajax Security, was published by Addison-Wesley in December 2007.


Access Control
Understanding Windows File And Registry Permissions
John R. Michener

This article discusses:
  • Access control lists
  • User rights
  • File system permissions
  • Registry and its permissions
This article uses the following technologies:
Windows Server 2008
Contents
Object Security Descriptors
Understanding Security Descriptor string_aces
Standard and Specific Rights
Integrity Labels and Their Usage
Object_guid and inherit_object_guid
Interpreting Security Descriptor string_aces
Windows Resource Protection
Setting Safe File System Permissions
Managing the Registry and Its Permissions
Wrap-Up
Whenever something happens in a system, a principal (which could be a process or thread acting on behalf of a user or service) acts upon objects. Files, directories, and registry keys are examples of commonly known objects. The basic security mechanism of Windows involves having a trusted system component check permissions and rights (AccessCheck) before an operation is allowed to proceed. Thus, you manage system behavior by setting permissions and rights. Since you cannot appropriately set permissions without understanding what is being done under the surface, I'll start by describing security settings on objects and how they are processed, and I'll follow that with how to set values for them.
Before I delve into the technical details, I want to take a look at the permissions at the root of the system drive in Windows Server 2008 using the Windows access control list (ACL) GUI. If I open Windows Explorer, select the security tab, right-click on Local Disc (C:), and select Properties, I see that administrators have full control. If I click on SYSTEM under Group or user names, I see that SYSTEM also has full control. When I click on Users under Group or user names, I see that the permission situation is not as simple: the users group on the system in Figure 1 has Read and Execute, List, Read, and so on. Clicking the Advanced button gives a more detailed view of the permissions associated with the users group (see Figure 2).
Figure 1 User Rights of Local Disk C
Figure 2 Advanced View of User Rights over Drive C
Here, members of the users group are allowed to create folders and append data to files in the root of the system drive. If you click the Edit button, you'll see another "special" grant to subfolders, shown in Figure 3. This operation requires administrator privileges.
Figure 3 Edit View of User Special Rights
Thus you see that normal users are allowed, by default, to create subfolders and add content to these folders from the root of the system drive in Windows Server 2008. This functionality was provided to members of the users group on Windows Server 2008 because some third-party software assumes that these permissions are present, and Microsoft did not want to break app compatibility.
Now let's move to a technical discussion of these issues and how they work below the GUI interface presented to the user. All named objects in Windows have security descriptors, which provide information about their owner as well as list which users and subjects have specified permissions. They also can specify which object accesses must be logged to the system event log.
The information about what a subject (user, process, and so on) is allowed to do to an object or resource is specified in a data structure known as an ACL. ACLs enumerate who (which principal) has what kind of access to specific objects. A discretionary ACL (DACL) is a type of ACL where the owners of objects are allowed to change them. Whenever an object is accessed, the security descriptor is compared to the principal's permissions to verify that the requested access is allowed.
Note that Windows also supports system ACLs (SACLs) for objects and has used SACL settings to establish which events are logged to the audit log for many releases. With Windows Server 2008 and Windows Vista, SACL has been extended to carry integrity-level information.
This integrity label is used to establish the "low" label that marks the Internet Explorer process used in LowRights Internet Explorer. "System" and "high" labels are used to protect critical system resources. The Windows message pump filters messages based upon the integrity level of the message. For example, medium-level processes do not receive messages sent from low-level processes, and high-level processes do not receive messages from low- or medium-level processes.
At this point, the integrity-level protection is a speed bump, not a true security barrier about which you can make security guarantees. The height of this bump will increase significantly in later releases when it is likely to become a real security barrier.
As with other modern OSs, Windows relies on DACLs for general access-control decisions. Here I will focus primarily on DACLs. For the system to determine whether a principal is allowed to perform an operation upon an object, several things are checked: the principal's privileges, the principal's token, and the object's security descriptor. The binary security descriptor on an object is passed to the AccessCheck routine with the principal's token. A requested access mask bit vector is prepared, representing the access rights that must be granted for the access check to succeed. It is passed with the principal's security descriptor to the AccessCheck routine, which examines the user's security token and considers the principal's privileges (typically based on roles or membership, such as administrator) in combination with the requested access and the DACL on the object.
If the requested access is satisfied by the principal's privileges, access is granted. Otherwise, the DACL access control entries (ACEs) are examined in order. As soon as the security system is able to show that all requested access components are allowed or that any of them is denied, it returns a success in the former case and a failure in the latter.
Thus, the DACL's list of ACEs should be appropriately ordered. The standard (canonical) ordering is to first place explicit denies, then explicit allows, general (group) denies, and group allows. If the canonical ordering isn't used, unanticipated allows or denies may occur.

Object Security Descriptors
While the security descriptor is a binary data structure, it relies on the security descriptor string format to provide a somewhat human-readable text format. A string format security descriptor is represented as a null-terminated string with tokens to indicate each of the four main components: owner (O:), primary group (G:), DACL (D:), and SACL (S:), as Figure 4 illustrates.
Figure 4 Security Descriptors
Security Identifiers (SIDs) are structured to provide parsing information and include 96 bits of random information (and may include 32 bits of sequence count) to serve as a unique identifier for owners. The string_aces (ACEs expressed in string format) are the structures that explicitly grant or deny permissions in the DACL and set policy in the SACL. Each string_ace is enclosed in parentheses and has the following structure:
(ace_type;ace_flags;rights;object_guid;inherit_
   object_guid;account_sid)
Only those grants that are necessary for proper access to the object in question must be present. Frequently the owner_sid and group_sid are omitted from the security descriptor. If not explicitly specified at creation time, the owner field of the security descriptor is set to the SID of the principal invoking the object creation. The group field is set to the primary group of the principal's security token. If it is not necessary to audit an object or to set an integrity label, the SACL will not be present.
Within the string_aces, only those fields that are necessary are included (the minimal set is ace_type, rights, and the subject, typically the account_sid). Typically, the object_guid and inherit_object_guid are not present. The system parses ACEs in order, from first to last, until access is either granted or denied. Thus, ordering of ACEs is important. "Deny permissions" should be placed before "allow permissions."
An ACL with no ACEs in it is an empty DACL. Since an ACE grants a specified subject access to an object, no one can access an object with an empty DACL. An object without a DACL is said to have a NULL DACL. Objects with NULL DACLs have not been secured and everybody has full control over them. For that reason, do not set either empty or NULL DACLs.
It is worthwhile now to look at what a realistic security descriptor looks like. Here's a security descriptor for the root of the Windows Server 2008 system drive (note that cacls is a legacy command-line routine for investigating and setting ACLs and is being replaced by icacls. Unfortunately, icacls does not support a command-line switch to output the results in standard Security Descriptor Definition Language, or SDDL, a switch that cacls has—the /S flag):
C:\>cacls c:/ /s
c:\"D:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICI;0x1200a9;;;BU)(A;CI;LC;;;BU)(A;CIIO;DC;;;BU)(A;OICIIO;GA;;;CO)"
Based upon what we know about security descriptors, you can see from the leading "D:" that no ownership or group membership is claimed and that the descriptor is a DACL. The DACL is protected: the "P" and the Windows NT 5.0 inherit flag is set. Then we have a number of ace_strings that have to be deciphered.

Understanding Security Descriptor string_aces
Recall the string_aces format I showed you previously. The allowed ace_types are defined in Figure 5, and the allowed ace_flags are defined in Figure 6. The ace_flags for inheritance are the ruling factors for the inheritance of ACEs.
Figure 5 Allowed ace_types
Figure 6 Allowed ace_flags
ACE rights are indicated by a string that denotes the access rights controlled by ACE. This string can be a hexadecimal string representation of the access rights, such as "0x7800003F", or it can be a concatenation of the Rights strings, such as "CCLCSWLOCRRC", which I will interpret later. The hex representation and associated bit values are shown in Figure 7.
Figure 7 ACL Access Mask
The system uses a single bitmap representation of ACE rights for all objects. Not all bits are meaningful for various objects. Only rights that are appropriate for an object are applied. Standard rights are those rights that are common to all securable objects. Generic rights are convenient shorthand for specifying rights of similar intent for various objects. The specification of generic rights is mapped into the appropriate set of specific rights. Integrity labels are also encoded using the ACE rights field when specifying the SACL. The available rights for various objects are listed in Figure 8.
Figure 8 Generic Rights
There are a number of largely equivalent rights mappings that are used rather indiscriminately. Full Control (FC) is equivalent to Generic_All (GA). For the file system, File All (FA) is the appropriate full control declaration. Key All (KA) is the appropriate full control declaration for the registry. Generic declarations are frequently used in place of the more appropriate declarations but are mapped to the appropriate file system or registry key declarations, as appropriate. SDDL expressions frequently mix these terms, thus you need to be aware of the equivalences.

Standard and Specific Rights
Many objects can be assigned rights. In addition to files and directories, we have registry keys, processes, desktops, and so forth. For the full list, see Figures A through J. Since we will be discussing permissions on the file system and registry, the specific rights for these objects are provided in Figures 9 and 10.
Figure 9 Specific File Rights
Figure 10 Specific Registry Rights

Integrity Labels and Their Usage
As was already stated, the integrity labels, if present, are stored in the object's SACL. Objects implicitly have medium integrity, so if there is no integrity label, the object has medium integrity. Similarly, if there is no integrity label on a security token, it also has medium integrity. The low-integrity label is used to label Low Rights processes, such as LowRights Internet Explorer and related untrusted objects. The "high" and "system" levels are used to help isolate those objects from medium and low processes and objects. The integrity labels are shown in Figure 11.
Figure 11 Integrity Labels

Object_guid and inherit_object_guid
Object_guid and inherit_object_guid are used in specifying the security of objects in Active Directory. They are not used when securing the file system or registry. "OA" and "OD" in the ace_type field of an ACE string correspond to an object-allow and object-deny ACE, respectively. In this case, the object_guid holds the guid of the object being permissioned and the inherit_object_guid holds the guid of the object from which it inherits permissions.
The account_sid field in the ACE structure denotes the security principal that is being granted or denied the access rights specified in the ACE. The account_sid field may hold a SID, which is a long structured identifier that is essentially meaningless to a person, or a shorthand "SID string" notation for a common account. The SID string notation for common accounts is used wherever possible to make the system more readable. A table of common or well-known accounts and their SID strings is shown in Figure J below.
The OW declaration of Owner Rights is new with Windows Server 2008 and Windows Vista. Previously, the Creator/Owner (CO) of an object had the standard rights of read control (RC) and Write_DAC (WD) over that object, enabling the owner to set the security on the object. This creates a problem if a user is a member of a group and creates a large number of objects. If that user leaves the group, he would still have control over those objects because he is the owner of these objects, which granted them RC plus Write_DAC permissions. The presence of the OW owner ACE restriction blocks the implicit grant of RC/WD to the owner unless these grants are explicitly made to the owner ACE of any other relevant ACE in the ACL. This allows mitigation of this security issue.

Interpreting Security Descriptor string_aces
The output from cacls for the root of the system drive was:
"D:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;BA)(A;OICI;0x1200a9;;;BU)(A;CI;LC;;;BU)(A;CIIO;DC;;;BU)
    (A;OICIIO;GA;;;CO)"
Parsing this for readability, you get:
"D:PAI
(A;OICI;FA;;;SY)
(A;OICI;FA;;;BA)
(A;OICI;0x1200a9;;;BU)(A;CI;LC;;;BU)(A;CIIO;DC;;;BU)
(A;OICIIO;GA;;;CO)"
This is a protected DACL with the auto-inherit flag for a modern file systems set. The protected flag means that inheritable parent grants won't be inherited; the DACL is protected from inheritance from the object's parent. In this case there is no parent, as it is the root.
The built-in administrator and system are granted inheritable File All over both files (due to the object inherit) and directories (due to the container inherit, or CI). This means that this DACL grants File All recursively on all files and directories below the root—except where inheritance is stopped by a protected DACL when the grant in the protected DACL must be examined. CO is granted Generic_All, which maps to File All, over both files and directories below the root directory (due to the inherit-only flag).
The grant to the built-in user is far more interesting. The first string_ace applies to both files and directories in the root and below and allows List, Read, ReadEA, Traverse, Execute, ReadAttr, ReadControl, and Sync. The second string_ace allows AddSubDir in the root and below (due to the IO—inherit-only flag), while the third string_ace allows AddFile in the directories below the root. This is the same as you saw when you explored these permissions with the ACL graphical interface of Windows Explorer.

Windows Resource Protection
Starting with Windows Server 2008 and Windows Vista, components declare their needed security settings in their manifests, which are signed by a Microsoft code signing root. The manifest specifies the ACLs and other permissions associated with the file. Thus, when a component is installed, it carries with it the appropriate security settings. In addition, OS files are protected from inadvertent damage by the system administrator using Windows Resource Protection (WRP). WRP relies upon a new system-level entity, Trusted Installer, to own and manage system files and folders.
A good facility to allow normal users to perform installations of authorized components was added in Windows Vista. Thus the Power User role is no longer required, and instances of ACEs that contain the Power User SID were removed. The Power User group still exists, but the component manifests have been scanned, and all detected instances of grants to PU have been deleted.
Let's look at a system directory to see the new permissions. This is also another good exercise in SDDL reading:
C:\>cacls c:\windows /s
C:\Windows "D:PAI(A;;FA;;;S-1-5-80-956008885-3418522649-18310
38044-1853292631-2271478464)(A;CIIO;GA;;;S-1-5-80-956008885-3
418522649-1831038044-1853292631-2271478464)(A;;0x1301bf;;;SY}
(A;OICIIO;GA;;;SY)(A;;0x1301bf;;;BA)(A;OICIIO;GA;;;BA)(A;;0x1200a9;;;BU)
(A;OICIIO;GXGR;;;BU)(A;OICIIO;GA;;;CO)"
Trusted Installer's SID is S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464. Using TI as shorthand, we find the following:
C:\Windows "D:PAI
(A;;FA;;;TI)(A;CIIO;GA;;;TI)
(A;;0x1301bf;;;SY)(A;OICIIO;GA;;;SY)
(A;;0x1301bf;;;BA)(A;OICIIO;GA;;;BA)
(A;;0x1200a9;;;BU)(A;OICIIO;GXGR;;;BU)
(A;OICIIO;GA;;;CO)"
Interpreting this, you see it is a protected ACE that is being applied to C:\Windows using the Windows NT 5.0 inheritance model.
Trusted Installer is granted full control over C:\Windows and is granted Generic_All over all child containers under C:\Windows (since it is CI, inherit only).
System and admin are granted Read, Write, Append, ReadEA, WriteEA, Execute, ReadAttr, WriteAttr, Del, RCtl, and Sync, = SDGRGWGX on C:\Windows. This is Generic_All minus Write_Owner and Write_DAC; Admin and System are granted everything but the ability to change the owner or the ACL. "BA" and "SY" have Generic_All over child file and directory objects.
Since the admin has the take ownership privilege, he can still assert WriteOwnership and take control anyway. Administrator and system are a security equivalent. But with this privilege, a determined administrator can circumvent the WRP ACL controls.
CO also has Generic_All over child file and directory objects. Built-in user has Read, ReadEA, Execute, ReadAttr, RCtl, and Sync, = GRGX over C:\Windows and GRGX over subdirectories and their files below C:\Windows.
All system files and folders have protected ACLs that grant Trusted Installer full control. The control of files by Trusted Installer is not expressed in the declaration on the system root directory but in the separate declarations of the Windows components.

Setting Safe File System Permissions
Now that you have some idea of how file system ACLs work and how to read them, let's look at setting them. If you are installing an application, you should install this to the default Program Files location. The default ACLs for this location give full control to Administrator and Local System, which is appropriate.
If you install an application to some other location or grant the user the ability to choose his preferred location for an application, you have a problem: the default ACLs for other drives and for non-system and non-application areas of the system drive are not secure enough. In such cases, you must explicitly ACL your folders with the appropriate protected DACLs. The simplest and safest choice for installing an application is to duplicate the security settings on the Program Files folder. If you choose not to do this, set the DACL so that non-administrators cannot change DACLs or ownership of executables and cannot write, append, or delete files in directories containing executables.
The basic rule if you are setting DACLs is that you do not want administrators or other users executing code that was written by a user. This is particularly a problem if the folder in question would be presumed to be trusted, typically by being in a trusted area (Windows, Program Files, and so forth). Doing so allows Elevation of Privilege (EoP) to administrator and increases the risk of cross-user attacks. Thus, if a user can write files to such a folder, other users and administrators should not be able to execute them.
At first glance, it would appear that you should not let users write to folders in Windows, System, Program Files, and so on, at any time. It turns out there are valid reasons for doing so. The most common is to record error log data. If the executable is running under the user's credentials and has to log an error, it needs write/append access to the error log/logging folder. (If you logged errors to per-user locations on a multi-user system, you would have the logging data spread over the system instead of being associated with the executable. Applications and services typically write to a shared folder or registry key.) You'll find the same issues in the registry, where error information is frequently stored in specified machine registry keys by a process running with user permissions.
Do not mix user writeable files with executable files. Use separate directories for files that must be trusted (such as executables) and files that must not be trusted (anything potentially written by an untrusted user). ACL the directories appropriately—administrator control over the executables and user read/execute, but not write. The same guidance applies to registry keys and subkeys.
Client and server differ here: server administrators are assumed to be far more knowledgeable than users running as administrator. Server administrators know that they must not execute code from untrusted areas of the system. By establishing a naming convention to isolate data files from executables in the system, you provide the administrator guidance concerning the trustworthiness of the executable—logging subdirectories are not trustworthy, thus users must not have the ability to create or modify these subdirectories, as this would enable them to spoof safe naming rules.
In the client environment, it is far easier to fool the administrative user into executing code. With such naive administrators, directories that have user write must not allow administrator execute privileges in order to prevent a user from installing an executable and then fooling an administrative user into running it and compromising the system. For example, if your application or service needs to store log information that will be written to under user privileges, you should create a logging subdirectory to hold this data. This subdirectory should not allow admin execute. One way to do this is to add a leading explicit deny for file execute to everybody, such as D;OI;WP;;;WD. This prevents cross-user attacks in directories that allow file write or modify to users.
There are many cases where it is expected that users will share data (though we do not want users sharing and executing code from a shared area). For example, a home user (or his photo-viewing application) is likely to create a folder such as C:\Photos. The user expects to be able to allow multiple users to write to this folder and have multiple users edit the various photos in this folder. The default ACLs on the root of the system drive on Windows Vista support this. The other common sharing situation is a folder in which users place data for other users to read. Only the creator of the data is allowed to delete or modify the data, but other users may copy it and then edit the copy. This is a shared read scenario, and it is the default for the system drive on Windows Server 2008.
Consider the case where you choose to lock down the system drive and, specifically, ACL folders for sharing. You have to choose the ACLs that are appropriate for these two rather common scenarios. We want administrators to be able to manage the objects and we want to prevent security issues associated with execution of code in these folders (note that the ACEs here prevent even the owner from executing code from these folders).
Here's a Shared Read ACE:
D:P(D;OI;WP;;;WD)(A;OICI;FA;;;BA)(A;OICI;FA;;;SY)(A;OICI;FA;;;CO)(A;CI;0x1200af;;;AU)(A;OI;GR;;;AU)
And here is a Collaborative ACE:
D:P(D;OI;WP;;;WD)(A;OICI;FA;;;BA)(A;OICI;FA;;;SY)(A;OICI;FA;;;CO)(A;OICI;SDGRGW;;;AU)
Note that both ACLs start with a deny execute ACE for everybody, object inherit (to apply it to files), to prevent user system and cross-user attacks. Then administrator, system (system does not really need it), and CO are given full control = File All.
For the Shared Read scenario, an authenticated user is granted List, AddFile, AddSubDir, ReadEA, Traverse, ReadAttr, RCtl, and Sync due to the restriction of this grant to CI, and thus to directories, and is separately granted Generic Read to all files. For the Collaborative scenario, an authenticated user is granted Delete, Generic Read, and Generic Write on files and directories.

Managing the Registry and Its Permissions
Windows stores much of its state information in the Windows Registry. Registry data stores are known as Hives, where data is stored in keys and subkeys, which are both viewed as containers (subkeys are not viewed as objects).
User-specific data is stored in the appropriate user section of Hive Key Users (HKEY_USERS). As one would expect, much of this data is writeable by the user. In any session, HKey_Current_User (HKCU) points to the appropriate section of HKEY_USERS.
System and machine information is stored in the HKEY_LOCAL_MACHINE (HKLM) hive. Included in HKLM is information for all the various system services, most of which now run with limited permissions under either the various Local Service or Network Service groups. Services and applications can store state information in their registry keys. This information should be stored in subkeys, either in the service key or in a key under the service key. The service key must not be ACL'd to enable the service to have SetKey over its own service key (or the WDac or WOwn, which would enable such an attack), as this allows the service to point to a different executable. Such an error introduces a potential EoP against the service host, as the Service Control Manager will load the executable that is pointed to when the system loads.
The general guidance for DACLs for HKLM is that they must not enable users to write or modify this data or the associated ACLs and ownership. As with the guidance on setting file system DACLs in system areas, exceptions occur for error logging where an app or a service running under a user or limited context needs to record error information. The guidance for such situations is similar to equivalent issues in the file system—create separate keys for such information and ACL them appropriately. Thus the sensitive information can be ACL'd to trusted subjects (administrator, system, and so on) and the logging data can be writeable, as needed.
The situation you are trying to avoid is a user modifying trusted parameters (such as turning the antivirus or anti-malware service off) or tampering with a tool that users or administrators use. Let's assume that when Notepad is invoked it loads C:\windows\notepad.exe. The default ACL over C:\windows does not allow an attacker to modify the executable. If the attacker can rewrite the link from the Notepad icon to its executable, the attacker can cause a different file, say C:\tools\load_rootkit.exe, to load. This could load a rootkit and then load Notepad so that the user would not be aware of the compromise.
If the attacker can drive the link through the registry, the protective ACLs on the file system are immaterial. You are concerned with attacks from limited system services against other system services as well. In Windows Vista and Windows Server 2008, services are separated into groups by the privileges they need. The defense-in-depth protections offered by this service isolation require configuration of the service permissions so that services cannot tamper with one another, particularly across service groups.
Just as we are concerned with preventing users from adding or linking to malicious executables, we must also prevent services from having the ability to change their permissions and capabilities. The ChangeConf privilege on services must be restricted to administrator, system, or Trusted Installer since this privilege allows the possessor to change the permissions on the service.

Wrap-Up
Windows provides a very rich set of permission controls that can be used to permit operations, block operations, and provide defense-in-depth against new threats. Unavoidably associated with this rich ability to control access is the issue of complexity.
Following a few general guidelines will help you avoid problems. For instance, the system defaults are reasonable compromises. You should use them. If you are installing an application outside of program files, use the program files ACLs. In some cases you may want to tighten defaults a little bit, such as the default grants to users on drives; but remember that if you indeed do this, you must be prepared to look for and deal with potential application compatibility issues.
The most important guideline is that administrators or system accounts must not execute code or follow pointers to code that a user can write or modify. Almost as important is that users not execute code or follow pointers to code that another user can write or modify. These guidelines drive all of the security issues discussed here. If any changes you make follow these guidelines, you have avoided the most serious security issues.
For more information about access control components, see "Access Control Components" on MSDN. Information about the ace_string access mask components can be found in the MSDN "ACCESS_MASK" article, which includes pointers to specific rights for files, directories, registry keys, and shared sections. Additional information about restricted SIDs can be found in the MSDN "Restricted Tokens" article.

The Full List of Rights
Figure A Standard Rights
Figure B Specific Rights for Files and Directories
Figure C Specific Rights for File Map and Registry
Figure D Specific Rights for Service Control Manager and Services
Figure E Specific Rights for Processes and Threads
Figure F Specific Rights for WindowStations and Desktops
Figure G Specific Rights for Symbolic Links and Events
Figure H Specific Rights for Semaphores and Mutexes
Figure I Specific Rights for Pipes and Tokens
Figure J Common or Well-Known Accounts and Their SID Strings
John R. Michener is senior security program manager for Microsoft. He joined Windows Security at Microsoft almost 5 years ago. John has more than 20 years of experience in system security and has done three security startups. He is the cryptography and permissions expert for the Windows Software Assurance team. You can reach him at jmichene@microsoft.com.


Utility Spotlight
12 Steps To Faster Web Pages With Visual Round Trip Analyzer
Jim Pierson

This article is based on a prerelease version of VRTA. All information is subject to change.

Code download available at: Visual Round Trip Analyzer (654 KB)

This article discusses:
  • Anatomy of a round-trip
  • Tips for faster Web pages
  • Visual Round Trip Analyzer
This article uses the following technologies:
Visual Round Trip Analyzer
Contents
Location, Location, Location
12 Easy Steps
Using Visual Round Trip Analyzer
Final Thoughts
So many factors can affect the performance of a Web page—the distance between server and client, the size of the elements on the page, how the browser loads these elements, available bandwidth. Finding those bottlenecks and identifying the culprits is no easy task. But identifying the causes can yield some significant improvements.
In this article, I'll show you how to spot and remedy some common causes of poor performance. I'll also introduce you to Visual Round Trip Analyzer (VRTA), a tool that can distinguish among the various factors that cause performance problems and present them as a graphical map. Then I'll show you how to perform your analysis using VRTA. But before we begin, find out how much you already know by taking the speed quiz called "How Do You Improve Page Loading?" that is included in this article.

Location, Location, Location
Two of the biggest factors in page load delays are distance, measured as round-trip time (RTT), and the number of round-trips between the client and servers. A round-trip is a network packet sent by a client to a server, which then sends a response packet. Downloading a single Web page can require as few as 2 or 3 round-trips if the page is sparse, or many tens of round-trips for a rich appearance. Opening a TCP connection is one round-trip. Sending a Get and receiving the first 3KB is one round-trip. Sending Acknowledgements for more data is more round-trips.
For users who are located near the Web site data center, say less than several hundred miles away such as a user in Los Angeles connecting to a server in Silicon Valley, RTT is only 20ms (.020 seconds) and not a significant factor in page load time. There could be 50 round-trips back and forth between the client and server in these circumstances and the page-load time would still only have 1 second of network delay before any server time was added (50 RTs × 20ms = 1 sec).
But the story is much different for a user in Europe or Asia connecting to the same Silicon Valley server. Japan and England are currently 120ms in round-trip time from California at minimum. When you consider a server disk seek time for each request is usually less than 10ms, versus round-trips of hundreds of milliseconds repeated many times over, you'll see that distance and the number of round-trips are easily the most significant contributing factor in long page-load times.
To combat the impact of long round-trips there are three main solutions you can try: you can reduce the number of serial round-trips by doing more in parallel and getting rid of excess weight, you can reduce the round-trip-time by moving your location closer to the user, or you can reduce server time.
Visualizing the page download accurately, as it really flows, is essential to performance analysis. For example, it's difficult to know which parts of the page are serial and which are downloading in parallel. File downloads are typically depicted as a waterfall (as you see in Figure 1) with one file represented per row in the report. But such a depiction does not account for the fact that since HTTP1.1, TCP connections are reused. Keeping connections open and calling additional files on the existing connection reduces the number of round-trips, but it's hard to see this in the traditional waterfall illustration. For a more accurate picture of how a page really loads you need to add the TCP port as a dimension of the drawing.
Figure 1 HTTP1.0 Waterfall Illustration
Figure 2 shows files loading on seven TCP ports, represented by the grey horizontal lines, and the number of milliseconds that have elapsed. The colored bars are the files; red is HTML text, gold represents style sheets, tan represents JavaScript, and blue lines are images. Here a single port is opened first, and then two more ports are opened. After the first files (CSS) are loaded, more files (JavaScript and images) are loaded on the same ports.
Figure 2 HTTP1.1 Keep-Alives
VRTA sits on top of the Microsoft Network Monitor (known as NetMon) 3.2 packet analyzer, though you don't need to know how to use NetMon. In the latest version of this tool, now to be released publicly, the Global Performance team added visual details for files and packets. Figure 3 shows the details. Once a TCP connection is created, the pink shaded line shows the TCP response packet coming from the server. The grey line is the time-to-first-byte from the server and the stairs indicate the response packets ramping up using TCP slow-start.
Figure 3 Details for Files and Packets

12 Easy Steps
It's easy to assume a Web page is always downloading at full speed and that more bytes will only slow down the page. The reality, though, is that unless the user is near the datacenter, it's very likely the browser will not make use of all the available bandwidth. Here are 12 different steps you can take to alleviate the problem.
1. Open Enough Ports Both the browser and the application can limit the number of ports opened in parallel. Internet Explorer 7.0, as well as earlier versions, limited the number of HTTP1.1 ports to two per domain. So, for example, msn.com gets only two ports. If you host all of your static JavaScript, CSS, and images on a single domain, as in Figure 4, then the older browsers would still open only two ports. Spreading the content over multiple domains allows the browser to open more ports. Alternatively, browsers can be modified to open more ports. This is currently under consideration by the Internet Explorer team.
Pages often bottleneck on the download of a single file or worse on a series of single files, one after the other. Ideally, a page is loading six to eight files in parallel. Take a look at the bit rate chart in Figure 4. There you'll see very little traffic for the first three seconds. In Figure 5, the orange file at the top-left corner of the chart is an SSL handshake. Inside the rectangles are indications about the packet flow that I will discuss in more detail later. If you scan the figure from left to right and count the number of files actively downloading at any one time, you'll see it's not until three seconds into the download that more than one file loads concurrently. The SSL starts out before another two red HTML files are loaded in series, then finally six ports are used to load nine files.
Figure 4 Too Few Ports and Many Small Files
Figure 5 Concurrent Ports
2. Limit the Number of Small Files to Be Downloaded It's hard to get full utilization of bandwidth across a wide area network (WAN) when shipping many small files. Figure 4 illustrates this throughout the page load but especially after the tenth second. Each of these icon image files is less than 500 bytes. If the server is 200ms RTT from the user and each file is one round-trip, then only five of these files could be loaded per second (5 files/sec × 500 Bytes/file × 8 bits/Byte = 20Kbps per TCP port).
Consolidating small files into larger ones allows more bytes to be sent per round-trip. For images, this is called image clustering or using sprites. A single file is downloaded with many images positioned side-by-side, which are then cropped out and placed on the page. Here's an example of this concept at work: a while back when we shipped Windows 2000, we had reports that it took six hours to download the upgrade to Germany from Redmond. My analysis found that 30,000 little files were being transferred one at a time. After we zipped them all into a single file, the download completed in 35 minutes instead.
3. Load JavaScript Files outside of the JavaScript Engine Browsers running Internet Explorer 7, and earlier versions, block on JavaScript files. Normally they download files in what's called speculative mode as quickly as possible. But when encountering JavaScript, the browser drops out of this mode and focuses on downloading the JavaScript. If you were to view this situation in VRTA you would see that none of the JavaScript files overlap with other JavaScript files and very few files of other types are loading at the same time.
One workaround is to move the JavaScript files to the end, but that misses the point that there is plenty of bandwidth to use. A better solution is to use a document.write of the files so they load outside of the JavaScript engine in the browser, as you see here:
function AsyncLoad()
{
var l = arguments.length;
for (var i=0;i<l;i++)
{
document.write("<script src='" + arguments[i] + "'></" + "script>");
}
}
AsyncLoad(
"file1.js",
"file2.js",
"file3.js");
We are anticipating that Internet Explorer 8.0 will finally resolve this problem.
4. Turn on Keep-Alives Opening a TCP port uses one round-trip, and getting a file uses at least one round-trip. If you can keep the connection open for additional files, you'll reduce the number of round-trips and increase the effective throughput. Note in Figure 6 that there is only one port per TCP connection (the grey horizontal lines). The bit rate jumps only on the larger files. You can see inside of each file rectangle the step pattern from increasing response volume.
Figure 6 Keep-Alives
This is TCP's slow start algorithm ramping up, sending more bytes per successful round-trip. That's how larger files can send at faster rates. Reusing a TCP connection allows subsequent files to continue with the expanded window size instead of starting over. Keep-Alives are on by default in IIS and unless you are hosting only a single file, they should always be left on.
5. Identify Network Congestion It's important to differentiate between server and network delays. Network delays can often be seen by looking at the TCP connect time or by looking for retransmissions. In the delta column of Figure 7 you see the time between packets (in milliseconds). The TCP request to open a connection was the first packet sent by the client (frame 67 in Figure 7). Then 2914ms (2.9 seconds) later the client resent the request (frame 306 in the figure) since it did not get a response from the server. After the second request, the server did respond 110ms later (frame 307).
Figure 7 Time between Packets
It is interesting to note that retransmissions of TCP connect packets happen on an exponential back-off basis while data packets within the connection are fast retransmitted. So, failed connections will often take two or more seconds for the first retransmission while data packets might only see a few hundred milliseconds to timeout.
6. Increase Network Max Transmit Unit (MTU) or TCP Windows Size Packet sizes can be as large as 1,514 bytes over ethernet. Sometimes servers may have the MTU (the size of the packets) limited to smaller sizes. If you see large files with response packets significantly less than 1,500 bytes, then you should check this.
Another related issue is the TCP window size. Normally this is 16 or 32KB, depending on the OS. That means that TCP can increase the number of bytes it sends before having to wait for an acknowledgment. You can think of this as the number of bytes per round-trip, though that is an over simplification. TCP/HTTP usually starts out by sending a window of two packets in the first response, about 3KB of data. Then the server waits for an acknowledgment before sending another window. The window increases in size with each successful acknowledgment received. This allows TCP to judge how much bandwidth is available between the two devices.
Look for window sizes of less than 16KB coming from a server for extended periods, then change these back to the default. To find the window size, open NetMon and look at the "window size" of the server response packets.
7. Identify Server Congestion A slow server often shows up with a fast TCP connect time, a fast acknowledgment response at the TCP level, and then a slow Time-To-First-Byte data response packet. Think of the server as operating in layers (as in the Open System Interconnection, or OSI, model), each working independently of the other. The TCP layer is running at the kernel level or in the NIC card. It's very fast. When the request packet comes in, it forwards the data up to the HTTP layer, which may need to do a disk seek. The TCP layer can't send off an acknowledgment until the HTTP layer can respond (see Figure 8).
Figure 8 Slow Server
8. Check for Unnecessary Round-Trips Not everything that happens during a page download actually needs to happen. Some actions are performed or decisions made simply because the browser is not given adequate instructions, like how long can a file be cached.
9. Set Expiration Dates Static files such as JavaScript, CSS, images, and XMLs are cached by the browser for when the user loads the page next. The browser will look for an expiration date in the file's HTTP header to see if the file is still good. Unfortunately, many sites are not setting these dates. So the browser pulls the files from the cache, doesn't find a date, and sends a Get-if-Modified to the server. In most cases, the server responds with 304 Not-Modified, and the file is used from the cache.
Performant sites set their dates to three years in the future. Changing the file name or path, or adding an argument string, causes the browser to reload the file. Pause to consider how many new servers and how much bandwidth was acquired to handle Get requests for files the users already had.
Another benefit of expiration dates is how they affect Entity tags (Etags). Etags tend to cause performance problems. They are meant to uniquely identity files using a number generated by the server. In clusters of servers, each server will create a different number. For example, let's say a browser is sending a Get-If-Modified for a file it found in its cache. If there is only one server, the file will most likely match and a 304 Not-Modified error will be sent. But, if the user is accessing a larger server farm, it's most likely to hit a different server with a different entity tag, causing that server to reload the entire file to the browser.
Using VRTA, if you notice more than one JavaScript file getting a 200-OK from the server then the server is not recognizing the entity tag and will resend the entire file. Not only is the file reloaded, but often the browser will recognize that the file is the same as the one in the cache and will attempt to stop the server from completely resending it.
The browser does this by sending a Reset flag to close the TCP port. That, in turn, increases the number of round-trips because of the fact that a new port is needed to complete the download of the remaining files. A simple solution is to set the expiration date for the file so the browser can reuse the cached file without making a call to the server.
Quiz: How Do You Improve Page Loading?
Answer these Web performance questions before you read the article and again after you've finished it to see how much your own performance has improved!
1. How many ports should you have running concurrently?
a. As few as possible
b. Many on broadband and fewer on narrowband
2. Which is faster to download across a WAN?
a. Many small files
b. A few large files of the same aggregate weight
3. JavaScript only blocks other JavaScript files.
a. False
b. True
4. Are Keep-Alives on by default?
a. Yes
b. No
5. The default expiration date
a. Is blank
b. Is 30 days from the download date
6. How do you stop Etags from reloading the file?
a. Set an expiration date
b. Ask the user to wait for the file to reload
7. Compression of HTML, JavaScript, and CSS files often achieves what reduction ratio?
a. 1.5 to 1
b. 3 or more to 1
8. Image files are always compressed.
a. True
b. Flase

10. Think before You Redirect Site addresses change over time and redirects are often used to point users to new locations. However, these redirects chew up time and should be replaced. Consider whether the site could respond from the old URL instead.
11. Use Compression Most JavaScript, CSS, and HTML files can be compressed by as much as 4-to-1. XML compresses even more. The neat thing about these file types is that the server can compress them one time and then store the compressed version. This reduces CPU load on the server and volume of bits on the networks.
Dynamic files such as ASPX need to be compressed for every user on the fly, so they are usually compressed by external devices like load balancers. I'm often asked whether compression consumes a lot of time on the browser and whether all browsers handle compression. First, consider the time it takes to load a large file across the network. If VRTA ran every file into a compression utility and then showed the ratio of the new file to the old in the compressibility column, ratios of 1 or less would represent files that are already compressed. In that instance, JavaScript files on this site are taking one to five seconds to load.
Uncompressing these files would take milliseconds for most modern PCs. Browsers must send a "will accept gzip, deflate" statement in the Get request to tell the server it can handle compression.
Full-size images should always be reformatted when used as thumbnails. Also, it's a good idea to check out different formats for images to see which compresses the best—JPG, GIF, or PNG. Strangely, I have found images showing up as not compressed, so don't ever assume that all images are compressed properly.
Also, remember to remove white space from files because this can often yield a 20% reduction in size. Note that white space includes tabs, new lines, and especially remarks. There are several tools available to do this. Crunching and compression are both advised as the effect is cumulative.
12. Edit Your CSS For some reason, developers tend to spell out every font instance in their CSS files instead of using the cascading effect. If you read a CSS file and see the same words repeated over and over, it's not written in a performant way.
Using Visual Round Trip Analyzer
You know what to look for; now go back and look at your site. Download VRTA from the MSDN Magazine Web site and install it. You will be prompted to install NetMon as well if you do not have version 3.2 or later already installed. This will require admin privileges on the machine you are using.
After the install is complete, VRTA will start automatically. You'll also find the icon sitting on your desktop. Click the Start button to have VRTA turn on NetMon capturing. Now visit your Web site again. Do only one atomic transaction (known as a Task) at a time. Navigate to one page and let it render—that's one transaction. Stop VRTA to view it. Before clicking on the Next button, start VRTA again and repeat after it renders.
Next, open the folder on the left to open an existing capture file. Also use this to view all files. VRTA creates a text version of all files, called RTA.txt. The NetMon icon takes you directly to the capture file. Save, copy, paste, and print work as usual. You've already used the Start and Stop button. The Zoom buttons allow you to magnify the file visualization. This is really helpful for looking at the smaller details. Use the Select network to tell VRTA to listen to your wired LAN interface or your wireless. Set the scale using the maximum time dropdown.
There are three hover windows in the tool. The first is on the Main Chart tab where you can see the downloading files by port. Hover over any of the files to see more details about it. On the All Files tab, you'll see the files sorted in start sequence. Hover over the URI to see details about each file. On the Analysis tab, hover over any of the rules to see the exceptions list of files.
In addition to the Copy and Paste buttons, VRTA creates two files. The NetMon capture file contains the raw packet details. From this you can recreate all of the VRTA UIs. You can also drill down into the capture to go beyond what the expert tool is able to present. The RTA.txt file is a tabular format of the page-load-time statistics. You can open this in Microsoft Office Excel for easier reading.
While you are looking at different Web pages with the tool, I'd like to make sure you are aware of a few important points in Web Performance testing. If you are sitting practically on top of your servers when you test (meaning basically within a few hundred miles), but your users are around the world, then you need to either get out in the world or use a WAN simulator (emulator). These are software utilities that can modify the bit rate, RTT, and packet loss on the network you are using.
At Microsoft, we use a WAN simulator benchmark of 300/64Kbps bit rate, 300ms RTT for global content and 50ms RTT for local Content Delivery Networks. Our packet loss setting is 3%. This benchmark allows us to test across multiple versions of multiple applications and see their relative performance in the same environment.
Remember, a sample of one is just that—a sample of only one. Every capture is slightly different, and often they may be very different because of the changing ad content and congestion on the network or servers. I highly recommend that you do several samples until you are able to see a steady state.
When NetMon installs, it includes a command-line version called NM_cap.exe. Create a script to start NM_cap, navigate to a page, then after the render, stop NM_cap and save the file. Open and review using the VRTA UI.
You should also recreate the first-time user experience (called Page-Load-Time 1) by going to an "About:Blank" page and then deleting the cached files in Internet Explorer | Tools | Internet Options. We use the blank page because Internet Explorer will automatically reload the page you are viewing. Close the browser to force any TCP connections to close. Then open the browser to the page you are testing. For Page-Load-Time 2 (the second time user returning the next day) just close the browser and reopen it to the test page.
One final warning: beware of corporate proxy servers—these servers firewall corporate resources from malicious intruders from the Internet. However, they also distort your performance sample. Compression is often turned off which may mislead you to thinking that it's off at your source server. Proxies also mask Keep-Alives. It's best to test outside of corporate proxies, directly on the Internet.

Final Thoughts
The report generated by VRTA and the best practices you've read about here about represent what is known today. But this is not all there is to know. Drill down into the raw data that NetMon captures and look for new patterns and best practices for yourself.
Many thanks to a great team for helping me to get this new visualization tool created: Ron Oshima, Lucius Fleuchaus, Jason He, Vinod Mangalpally, and Doug Franklin.

Jim Pierson has been with Microsoft for 12 years, 5 of which were focused on Web Performance Engineering. Jim is Principal Group Manager in the GlobalPerf team that provides tools for measurement, analysis, and training of best practices to the online services of MSN and Windows Live.

Quiz answers: 1=b, 2=b, 3=a, 4=a, 5=a, 6=a, 7=b, 8=b


RIA
Light Up SharePoint With Silverlight 2 Web Parts
Steve Fox and Paul Stubbs

This article is based on a prerelease version of Silverlight 2. All information is subject to change.

This article discusses:
  • Creating a Silverlight media player
  • Creating the Web Part
  • Deploying to a SharePoint site
  • Troubleshooting the app
This article uses the following technologies:
Silverlight 2, SharePoint
Contents
Creating the Media Player
Coding the Application UI
The Application Code
Creating and Deploying the Web Part
Adding the Application to the SharePoint Site
Troubleshooting
In the world of Web-based application development, users are demanding ever-improved application experiences. Further, the demands of these Web-based applications go beyond supporting a single browser technology or platform. If you are building a Web app today, you need technologies that support not only a way to deliver a more compelling user experience, but also that enable developers to deliver this experience quickly to multiple technologies and audiences.
Silverlight 2 is one example of a technology that enables robust Web application development opportunities. Silverlight is a cross-browser, cross-platform plug-in that provides developers access to a rich programming model for building Web-based user experiences.
SharePoint is another technology that is frequently being used for Web-based application development. Companies that invested in SharePoint for intranet development now want to extend this technology out to externally facing Web sites. Together, Silverlight and SharePoint provide a simple, yet powerful, infrastructure for building intranet and extranet applications with sophisticated user interface designs and interactions.
In this article, we'll discuss one of the key ways in which you can use SharePoint and Silverlight together by integrating a Silverlight application within a SharePoint Web Part. We start by walking you through building a fairly simple media application using Silverlight 2 Beta 2 (see Figure 1). We'll then show how you can use the SharePoint Web Part as a wrapper for your Silverlight application. This is an interesting approach because it enables you to not only create isolated hosted applications, but also interconnect the Web Parts through the native SharePoint object model—for example, through a provider/consumer model.
Figure 1 Silverlight WMV Player Web Part

Creating the Media Player
Silverlight 2, in beta 2 as we go to press, is quite an evolution from Silverlight 1.0. For example, Silverlight 2 supports managed code, so in addition to JavaScript, you can now use Visual Basic and C# in your applications. Further, Silverlight 2 ships with a set of controls and facilitates the extension of controls to make the job of creating Silverlight-based user interfaces much easier. The designer experience is now much better and is supported by both a UI designer tool within Visual Studio 2008 and Expression Blend.
To prepare your environment to build Silverlight applications for SharePoint, make sure you have the following installed:
  • Windows Server 2003 or Windows Server 2008
  • Windows SharePoint Services 3.0 and SP1
  • Microsoft Office SharePoint Server 2007 and SP1
  • Visual Studio 2008 Professional Edition
  • Silverlight Tools Beta 2 for Visual Studio 2008
  • Silverlight 2 Beta 2 Runtime
  • Windows SharePoint Services 3.0 Tools: Visual Studio Extensions, Version 1.2
After you've set up your environment, open Visual Studio 2008 and create a new project. In either the C# or Visual Basic nodes, you'll find a new project node called Silverlight, which contains two project templates: one to create a Silverlight application and another to create a Silverlight class library (see Figure 2).
Figure 2 Silverlight Application New Project Template
For the sample app, we selected the Silverlight application template, provided a name (MediaPlayer in our case), and accepted the other default options. After completing those steps and clicking OK, you will be prompted to either add a new Web to the solution or to dynamically generate an HTML test page for hosting Silverlight within the project. Select the first choice (adding a new Web to the solution), again accepting the default options. When finished, your solution will consist of both the Silverlight project and the hosting Web application (as opposed to just an HTML page).
In our example, you can see these two projects in the MediaPlayer solution. The first is a Web site called MediaPlayerWeb. This is the test Web site that you will use to develop and test your Silverlight application. The second project is the Silverlight application called MediaPlayer. For this article, we'll focus on the Page.xaml and the Page.xaml.cs files in the MediaPlayer project.
In Silverlight 2, the UI is generated using an XML-based declarative language called XAML—the same language used to create Windows Presentation Foundation (WPF) UI for client applications. Page.xaml contains the XAML code that defines the UI, while Page.xaml.cs contains the codebehind for the application.
You can have only one Page.xaml.cs file, but you can have multiple XAML files that can be managed through the central Page.xaml file. You might do this if you want to manage separate Silverlight controls in one Page.xaml.

Coding the Application UI
Visual Studio 2008 provides a split design/XAML view when you open XAML files. Additionally, in Silverlight 2 Beta 2, you can drag controls from the toolbox and drop them on the XAML view. (Dragging controls to the designer surface is not supported in Visual Studio 2008, although you can drag and drop to the designer surface in Expression Blend.)
The UI that we created for MediaPlayer is deliberately simple in order to illustrate the fundamental aspects of integration. Figure 3 shows the XAML code that represents the UI for the Silverlight application. We're using StackPanel elements to contain and position other controls within the UI. One hosts a MediaElement, the element that will play the WMV file and a Slider element that represents the timeline for the short movie. Another hosts Button, ToggleButton and CheckBox elements, which will provide playing, pausing, stopping, and muting functionality.
Figure 3 Media Player XAML
<UserControl x:Class="MediaPlayer.Page"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Width="400" Height="500">

    <StackPanel Grid.Row="1"
                Margin="20,10,20,10"
                Background="White"
                Visibility="Visible">

        <MediaElement x:Name="moviePlayer"
                      Source="Bear.wmv"
                      Margin="5">
        </MediaElement>

        <Slider x:Name="timeline"
                ValueChanged="timeline_ValueChanged"
                Maximum="1"
                Margin="5" >
        </Slider>
        <StackPanel Orientation="Horizontal"
                    HorizontalAlignment="Center">
            <Button x:Name="movieStop"
                    Content="Stop"
                    Height="25"
                    Width="40"

                    Click="movieStop_Click"
                    Margin="5">
            </Button>
            <ToggleButton x:Name="moviePlay"
                          Content="Play"
                          Height="25"
                          Width="40"

                          Click="moviePlay_Click"
                          Margin="5">
            </ToggleButton>
            <CheckBox x:Name="movieMute"
                      Content="Sound On"
                      Click="movieMute_Click">
            </CheckBox>
        </StackPanel>
    </StackPanel>
</UserControl>
One of the key properties that you'll want to take note of in the XAML-based UI is the x:Name property. This represents the name provided for the control, which you can use to manipulate it from the codebehind. Also note the events that correspond to some of the controls in the XAML, such as the button's click=moviePlay_Click, which will call the moviePlay_Click method in the codebehind when clicked.
In the XAML code, we configure the MediaElement control to play the movie Bear.wmv. You can load your own movie from the ClientBin folder, but you'll need to make sure you change the Source property in the XAML to reflect the name of the file you add to your project. Also note that you can provide an HTTP reference to a location of a WMV file, such as using a picture library as a repository for your movies in your SharePoint site.
Note that when creating the event handlers within the Page.xaml file for each of the event handlers (such as Click="movieMute_Click"), if you type Click and then use IntelliSense to add the event handlers, Visual Studio will automatically add the stub methods for your event handlers in the Page.xaml.cs file.
Before we move on to the codebehind, let's take a quick look at the Expression Blend view of this same code. We did all of the coding within Visual Studio 2008, but you can also use Expression Blend to create a UI that can subsequently be integrated into your Silverlight project in Visual Studio. After opening the page in Expression Blend (see Figure 4), you can then add items to the Silverlight application, save the file, and then switch back to Visual Studio 2008, which will prompt you to reload the changes from Expression Blend in your Visual Studio 2008 project. (To open in Expression, right-click Page.xaml and select Open in Expression Blend.)
Figure 4 Project View in Expression Blend

The Application Code
To add codebehind for the Silverlight application, right-click the Page.xaml file in the Solution Explorer and click View Code. You can also expand the Page.xaml node, right-click the Page.xaml.cs file, then select View Code. The code in this view corresponds to the UI we built earlier—this is where we set specific properties of the Silverlight controls. Figure 5 shows the code that we added to the Page.xaml.cs file.
Figure 5 Media Player Codebehind
namespace MediaPlayer {
    public partial class Page : UserControl {
        public Page() {
            InitializeComponent();
        }

        private void timeline_ValueChanged(
          object sender, RoutedPropertyChangedEventArgs<double> e) {
            long ticks = (long)((sender as Slider).Value *
              this.moviePlayer.NaturalDuration.TimeSpan.Ticks);
            TimeSpan movieTimespan = new TimeSpan(ticks);
            this.moviePlayer.Position = movieTimespan;
            if (this.moviePlayer.CurrentState == MediaElementState.Playing) {
                this.moviePlay.IsChecked = true;
                this.moviePlay.Content = "Pause";
            }
        }

        private void movieStop_Click(object sender, RoutedEventArgs e) {
            this.timeline.Value = 0;
            this.moviePlayer.Stop();
            this.moviePlay.Content = "Play";
        }
        private void moviePlay_Click(object sender, RoutedEventArgs e) {
            if (moviePlay.IsChecked == true) {
                moviePlay.Content = "Pause";
                this.moviePlayer.Play();
            }
            else {
                moviePlay.Content = "Play";
                this.moviePlayer.Pause();
            }
        }

        private void movieMute_Click(object sender, RoutedEventArgs e) {
            if (movieMute.IsChecked == true) {
                movieMute.Content = "Sound On";
                this.moviePlayer.IsMuted = true;
            }
            else {
                movieMute.Content = "Sound Off";
                this.moviePlayer.IsMuted = false;
            }
        }
    }
}
You'll note that in the codebehind there are a number of methods, all of which correspond to the event handlers that we added to the controls in the XAML code. For example, in this code the movieStop_Click method responds to the user clicking the Stop button, which then sets the timeline value to 0 (in other words, back to the start of the movie) and changes the text on the Play button (which changes from Pause to Play depending on what state it's originally in).
This codebehind also responds to the user clicking the Play/Pause toggle button, which calls the appropriate Play and Pause methods and sets the text on the toggle button. The user can also interact with the slider control, which will jump the movie forward to a specific point—the equivalent to a fast-forward/rewind functionality. And finally, the user can mute the movie by clicking the Mute checkbox.
At this point, you can debug the Silverlight application by pressing F5 on your keyboard or clicking Debug | Start Debugging from the Visual Studio project menu. This will invoke your default browser and then load the Silverlight application. You may see a warning that debugging is not enabled. Click OK to have Visual Studio modify the web.config file to enable debugging. If everything is working correctly, the WMV file (in our case, Bear.wmv) will load and begin playing. You can alter the play state by clicking Stop, Pause, or using the slider control. You can also affect the sound by checking the Mute checkbox.
While the Silverlight application is now working, you need to make sure you complete one more step before moving on to integrating it with SharePoint. This step is to ensure that the XAP file, the archive that contains your built Silverlight application, is located where the SharePoint Web Part can pick it up and load it.
The XAP file contains all of the core application files for your Silverlight application. Because the XAP file is essentially an archive of the built project files, you can rename the XAP file with a .zip extension and then open it up to see the files that are contained within the XAP package.
When you open up the MediaPlayer.xap file, you will see several different files: the core DLLs for the project and an AppManifest.xaml file. The DLLs in the archive are a combination of the code that you create (in this example, the DLL we created was MediaPlayer.dll) and Microsoft DLLs required to run your compiled assembly. The AppManifest.xaml file is generated and lists any dependencies from your project. Here are the contents of the AppManifest.xaml file from the MediaPlayer example discussed in this article, which contains a number of key references specific to the deployment of the Silverlight application:
<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  EntryPointAssembly="MediaPlayer"
  EntryPointType="MediaPlayer.App"
  RuntimeVersion="2.0.30523.6">
  <Deployment.Parts>
    <AssemblyPart x:Name="MediaPlayer" Source="MediaPlayer.dll" />
    <AssemblyPart x:Name="System.Windows.Controls.Extended"
      Source="System.Windows.Controls.Extended.dll" />
  </Deployment.Parts>
</Deployment>
It shows the name of the assembly (MediaPlayer), the Silverlight runtime version we used to compile our application (2.0.30523.6), and then lists out the required assemblies, which are also contained within the XAP archive.
Make sure that the XAP file is copied to a location within your SharePoint site. For the MediaPlayer example, we created a folder called ClientBin in the root virtual directory of our SharePoint site. You could alternatively copy the XAP file to a SharePoint document library or to the _layouts folder on your SharePoint server. Within ClientBin we created a folder called XAP, which contains all of the XAP files (or Silverlight applications we're building and integrating with SharePoint). You need to reference this location when creating an instance of the Silverlight control in the SharePoint Web Part project. You also need to make sure that when you update and recompile your Silverlight application, you don't forget to copy the most recent XAP file into the appropriate directory.
After you've copied your XAP file to a location on your server, you're now ready to move onto creating the SharePoint Web Part host container for your Silverlight application.
Renaming a Web Part
If you accepted the default name for your Web Part project, the project shell contains a Web Part called WebPart1. You could keep this, but you probably want a more descriptive name for your Web Part.
To do this, right-click the existing Web Part (WebPart1) and select Delete. Next click the WSP tab and delete everything under WebPart1WebPart. (If the WSP view—the view of the Web Part package as it will be deployed to SharePoint—is not displayed, from the main Visual Studio menu click View, Other Windows, and select WSP View.) After confirming the deletion of the Web Part, right-click the project in Solution Explorer and then select Add New Item. In the Add New Item dialog, select the SharePoint node and then select Web Part. Provide a more intuitive name for the Web Part.
Adding a New Web Part to the Project

Creating and Deploying the Web Part
To create the Web Part, open Visual Studio and open a new SharePoint Web Part project template (see Figure 6). In our example, we've named the new Web Part project MediaPlayerWebPart.
Figure 6 Web Part Project Template
After you've added your new Web Part, you'll need to add two references to the project before you can add code. The first reference is the System.Web.Extensions DLL, and the second is the System.Web.Silverlight DLL. To add code to the Web Part, right-click the Web Part code file (for example MediaPlayerWebPart.cs) and click View Code. Figure 7 shows the code that we added to the Web Part project.
There are two key items to call out. The first is the fact that we are using the ScriptManager object to render the Silverlight application. This is because the Silverlight application has a dependency on the AJAX ScriptManager control, so an instance of the ScriptManager object needs to first be added to the project.
The second is the instantiation of the Silverlight control. In the code you see in Figure 7, myMediaCtrl represents the instance of our Silverlight control (effectively the Silverlight media player we created). After you create a new instance of the control, you'll need to set a number of properties. Very important among these properties is the Source property, which points to the location of the XAP file and then loads that within the Web Part.
Figure 7 Media Player Web Part
namespace MediaPlayerWebPart {
    [Guid("7e7ef885-130a-45e6-ab14-15f344b1939d")]
    public class MediaPlayerWebPart :
      System.Web.UI.WebControls.WebParts.WebPart {
        public MediaPlayerWebPart() {
        }

        protected override void OnLoad(EventArgs e) {
            base.OnLoad(e);

            ScriptManager scriptManager = ScriptManager.GetCurrent(this.Page);
            if (scriptManager == null) {
                scriptManager = new ScriptManager();
                this.Controls.Add(scriptManager);
            }
        }

        protected override void CreateChildControls() {
            System.Web.UI.SilverlightControls.Silverlight myMediaCtrl =
              new System.Web.UI.SilverlightControls.Silverlight();
            myMediaCtrl.ID = "myMediaPlayer";
            myMediaCtrl.Source = "/ClientBin/XAP/MediaPlayer.xap";
            myMediaCtrl.Width = new System.Web.UI.WebControls.Unit(900);
            myMediaCtrl.Height = new System.Web.UI.WebControls.Unit(650);

            this.Controls.Add(myMediaCtrl);
        }
    }
}
With the code added, you are now ready to complete the second step: configuring the Web Part project for deployment. To do this, open the project's the Properties view and click the Debug tab. Check the Start Browser with URL checkbox, and then enter the root URL for your SharePoint site (for example, http://spvm).
To test the deployment of the Web Part, click Build, and then Deploy Solution. This automatically creates a WSP package (the CAB equivalent for SharePoint assembly deployments) that contains all of the files that are required to deploy the Web Part assembly to SharePoint, deploys the solution files to the appropriate places on your SharePoint site, and then stops and restarts the Internet Information Services (IIS) Web application.
If you are trying to deploy on Windows Server 2008 with UAC turned on, you need to run Visual Studio as Administrator or you will get an "Object reference not set to an instance of an object" error. Assuming everything was configured correctly, you'll now be able to add the Web Part to a site within SharePoint, thus the last step in your integration between SharePoint and Silverlight.

Adding the Application to the SharePoint Site
At this point, you've created the Silverlight application and you've created and deployed the Web Part container for the Silverlight application. The last remaining step is to create a site that can host the Web Part. Before you do this, though, there are a couple of items you will want to verify.
The first thing to verify is that you've installed System.Web.Silverlight.dll in the global assembly cache (GAC). If it is not, install it from the installation location \Program Files\Microsoft SDKs\Silverlight\v2.0\Libraries\Server.
The second thing to verify is that the web.config file in your SharePoint root directory has all of the necessary elements to support a Silverlight application and the Silverlight application is marked as a safe control. The following illustrates the safe control entry for the MediaPlayerWebPart in the web.config file (which is automatically added when you build and deploy your project to your SharePoint site):
<SafeControl Assembly="MediaPlayerWebPart,
             Version=1.0.0.0,
             Culture=neutral,
             PublicKeyToken=9f4da00116c38ec5"
             Namespace="MediaPlayerWebPart"
             TypeName="MediaPlayerWebPart"
             Safe="True" />
While we won't talk exhaustively about the web.config file, when you're trying to make sure that the web.config file in the root directory of the SharePoint site supports the Silverlight application, the easiest way to do this is to review the web.config file that is built as a part of the Silverlight project and make sure all of those elements exist in the web.config that resides in the root folder of your SharePoint site. If all of these elements are not present, your Silverlight application will not appear in the SharePoint Web Part.
You are now ready to create the SharePoint site that will host the Silverlight-based Web Part. To do this, navigate to your SharePoint site. We created a separate test site for our Silverlight application called Media Player. To create a site, navigate to the home page of your SharePoint site and click Site Actions | Site Settings. Under Site Administration, click Sites and Workspaces. Within the Sites and Workspaces page, click Create. Provide a name, description, and type of site (we selected Blank site) and URL for your site, and then click Create.
When the new site is created, SharePoint will automatically load it for you. Click Site Actions and Edit Page. In the site, click Add a Web Part, which invokes the Web Part Gallery (see Figure 8). Scroll down until you find the Web Part project you added to SharePoint (for example MediaPlayerWebPart), select the checkbox, and then click Add. This will add the Silverlight media player application into SharePoint by way of the Web Part as a host container. The Silverlight application will respond the same as when you were debugging it within Visual Studio 2008—it just appears as a Web Part within SharePoint. After you've added the Web Part to SharePoint, try out the different controls within the Silverlight application to make sure they all work.
Figure 8 Web Part Gallery
Further Reading on Web Parts

SharePoint and Silverlight Blueprints
ssblueprints.net/sharepoint

Silverlight Home Page
silverlight.net

SharePoint Development Resources
MSSharePointDeveloper.com

Troubleshooting
After you've added the Web Part to SharePoint, be sure to try out the different controls within the Silverlight application to make sure they all work. As you know, development is not always without some degree of troubleshooting, so we'll offer a couple of things to look out for as you integrate your Silverlight applications with SharePoint.
If you get network errors when loading media, this is often because the deployed assembly cannot find the media. If the media path is relative, it is relative to the XAP location, so make sure the media loads properly in the Silverlight application.
Make sure the XAP Source location is correct within the Web Part file. Many times, we've seen the case where the Web Part loads without any Silverlight application. This is usually because the Web Part cannot find the XAP file. Also, when you update your Silverlight application, make sure to recopy the newest version of the XAP file into the source location that you've specified in the Web Part project.
If you have successfully added the Web Part, but the Silverlight still does not show up, then you should double-check the web.config file that is in your root SharePoint directory and make sure it not only has the Web Part marked as a safe control (as discussed earlier) but also has all of the necessary elements in order to support the Silverlight application.

Steve Fox is a Technical Evangelist with the Developer Platform Evangelism (DPE) team at Microsoft. He spends most of his time evangelizing Office and SharePoint development and working with companies that want to use them in their development efforts. Steve has coauthored a number of books including Professional Microsoft SharePoint Development Using Silverlight 2. You can check out his blog at blogs.msdn.com/steve_fox.

Paul Stubbs is an Architect in the Platform Architecture Team, where he focuses on solutions architecture with an emphasis on information worker productivity, business-to-customer solutions, and Web 2.0 social networking. He has authored two books on solution development using Microsoft Office, several articles for MSDN Magazine, and has also spoken at Microsoft Tech•Ed and Tech Ready conferences. Read Paul's blog at blogs.msdn.com/pstubbs.


Editor's Note
Can I See Some Identification?
Howard Dierking

Recently my wife and I went to see a movie, and I had to purchase our tickets from a ticket agent who, as a result of an intercom system designed to transmit through half-inch-thick glass, sounded like a synthesized voice run through a distorted amp. And imagine my surprise when I was asked for my ID because the movie was rated R.
For anyone unfamiliar with the movie rating scale in the U.S., an R means individuals younger than 17 are not allowed to attend unless accompanied by an adult. Now, I realize that I probably look a bit younger than I am. But under 17? I showed my ID and purchased my movie tickets, but the incident caused me to think about some broader issues related to security requirements and context.
One realization has to do with the thick sheet of glass separating me from the agent. I suspect that the actual reason for it is based in history, when most moviegoers used to purchase tickets using cash. And while the rise of credit cards may make it less common today, those folks behind the window still have to manage a lot of cash. Based on where most ticket counters are located relative to the rest of the theater, without that window (and the distorted intercom), the ticket agent would be extremely vulnerable to anyone with a broken moral compass and a desire for the cash.
Another case where different contexts yield different security requirements is in age validation. If I'm buying tickets at the counter, I'm a captive audience—the agent can check my ID and verify my age with minimal effort or impact to the process of selling a ticket.
But if I'm buying tickets online, while the business process remains consistent, the change in context shifts the security focus from protecting the assets of the movie theater (cash) to protecting the assets of the purchaser (credit card number). In a context where the ticket purchaser is essentially anonymous, deeper validation steps like ascertaining the purchaser's age may seem superficial.
As I've said many times, the number of technologies that must be integrated into our apps and enterprises is enormous, and it keeps growing. Additionally, each change in technology causes a change in the security context, potentially altering the security requirements. I have seen many projects start out with the simple goal of moving an existing app to the Web, only to later require a great deal of additional effort (or stall out completely) because the impact of the context change was not planned for up front.
From the technology perspective, there's no simple solution here. Security measures must be designed and implemented based on a proven understanding of the system, the users, and the context in which the system will be run. Given that, I firmly believe building secure systems has more to do with people and processes.
We focus a lot of attention on the security development lifecycle (SDL). This is meant to underscore the idea that security is contextual and must be a standing discipline along with other major phases of a development lifecycle. As you will see in this issue, the SDL has also undergone some changes to remain achievable even in high-change environments where agile development methodologies are most effective. In fact, I guess you could say that the SDL is changing in response to a change in context.

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Paul Andrew, Jonathan Aneja, Bob Brumfield, Jeff Cao, Kiran Dowluru, Ryan Duguid, Mike Flasko, Matt Gibbs, Kevin Gjerstad, Christian Kleinerman, Bertrand Le Roy, Varsha Mahadevan, Duane Need, Wayne Norton, Eugene Osovetsky, Dave Reed, Michael Rys, Gerhard Schneider, and Michael Wang.

It is with great sorrow that we note the passing of Paul DiLascia, a long-time contributor to MSJ and MSDN Magazine and a friend to many of the staff here at the magazine. He shall be missed.


Toolbox
Online Scheduling, Jeff Smith’s SQL Blog, Easy Regular Expressions, And More
Scott Mitchell

All prices were confirmed September 18, 2008, and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions at Microsoft.

Contents
Create an Online Calendar and Scheduling Interface
Blogs of Note
Your Own Regular Expression Assistant
The Bookshelf

Create an Online Calendar and Scheduling Interface
Many business applications require users to be able to easily assign assets to resources for a particular date and time. For example, the software that powers a doctor's office needs to schedule assets such as doctors, patients, and equipment to resources such as offices and rooms. And a time sheet application associates employees' time and activities with projects and meetings.
Although ASP.NET includes a Calendar Web control, it lacks the features and functionality needed for a robust scheduling user interface. The good news is that implementing a professional-looking and feature-rich scheduling experience is a walk in the park with DayPilot 4.8 by Dan Letecky. DayPilot is a collection of Web controls that display AJAX-enabled calendar and scheduling user interfaces. Out of the box, DayPilot lets you view the schedule for a particular asset using month, week, or day views. These user interfaces mirror those found in desktop-based scheduling applications, such as the calendar in Microsoft Outlook.
There is also a scheduler view, which lists resources along the left column and dates and times across the top. This view makes it easy to see how the resources are allocated and to spot conflicts and available resources.
The information displayed in the DayPilot controls can be specified in a number of ways. You can programmatically supply this information as an array; if the data resides in an XML file or database, you can use a data source control or programmatically bind the data. Once the data is displayed, you can customize the appearance using a variety of properties. What's more, there are many events and extension points, making it easy to customize the display and behavior of these controls. For example, it takes just a few lines of code to instruct DayPilot to use different background colors for an event based on the asset or resource.
The most exciting aspect of DayPilot is its integrated AJAX support. The DayPilot user interfaces let the end user drag and drop scheduled events and to modify event durations and information by clicking, dragging, and double-clicking. There is also an integrated context menu, which is displayed when the user right-clicks on an existing scheduled event. The context menu can be used to modify the selected event, delete it, or perform some custom action.
At the time of this writing, DayPilot has been thoroughly tested with Internet Explorer 6.0 and up and Firefox 2.0 and up. The development roadmap includes upcoming support for both the Safari and Opera browsers.
DayPilot is available in two editions: an open-source version, which is free, and a Pro version. In a nutshell, the open-source version provides read-only day and week views. To enjoy the month and scheduler views and to allow your users to move and edit events from the DayPilot user interfaces, you will need to use the Pro version.
Price: Free open-source version; $549 for the Pro version.
daypilot.org/daypilot-pro-4-8.htm
DayPilot Makes Implementing a Feature-Rich Scheduling Experience Simple

Blogs of Note
Most application developers have a working knowledge of SQL, the query language used to retrieve and modify information from a database. They know how to perform the most common tasks performed by data-driven applications—selecting data, filtering and ordering the results, and updating, deleting, and inserting records—but they get stuck when faced with a data-related challenge that falls outside the ordinary. To help boost your SQL skills, check out Jeff Smith's blog. Jeff is a .NET developer who's got a knack for SQL. He posts an assortment of tips and tricks every week, along with articles and guidance on database design.
Jeff's blog entries explore topics such as what the ALL keyword does in the GROUP BY statement; how left and right outer joins differ and whether one is better than the other; how to retrieve the top n percent per group; and how to best choose column data types when designing your database. And each blog entry is written in a down-to-earth style that is easy to read and understand.
Do you have a particularly tricky query that's got you stumped? Send in your question to Jeff. Don't worry, you won't be imposing, as Jeff encourages his readers to e-mail him their SQL-related questions. These questions and their answers are then posted to Jeff's blog so that everyone benefits. Even if you don't have a question (or are too shy to ask one), you can still learn a lot by reading other people's questions and Jeff's solutions.
weblogs.sqlteam.com/jeffs

Your Own Regular Expression Assistant
Regular expressions are a terse syntax for matching string patterns and are an ideal way to search for or replace strings in a body of text. They are commonly used in validating user inputs that must conform to a particular pattern (telephone numbers, e-mail addresses, and so on) and in advanced search and replace scenarios.
Short and simple regular expressions are usually easy to write and read, but longer and more complex ones can be notoriously difficult to craft. A lengthy regular expression, with its conglomeration of literal characters, special symbols, groups, captures, and lookaround functionality, can quickly morph into an unintelligible blob of characters. For help creating, parsing, testing, and debugging regular expressions, check out RegexBuddy 3.1.1 by Just Great Software.
With its Insert Token toolbar, RegexBuddy makes creating a regular expression as simple as point and click. Choose what tokens to add to your regular expression: literal text; character classes such as digits, white space, or alphabetic characters; capturing or non-capturing groups; back references; and so forth. RegexBuddy shows the regular expression as text and as a tree of tokens. Adding a new token from the Insert Token toolbar adds the appropriate token to the tree and updates the regular expression. Selecting a token from the tree highlights the corresponding characters in the regular expression and vice versa.
This token tree view is very helpful in understanding an existing unwieldy regular expression. Paste the regular expression into RegexBuddy and the token tree is automatically loaded, explaining the behavior of each token in the regular expression step by step.
RegexBuddy provides tools for testing and debugging your regular expressions. After creating or entering your regular expression, go to the Test tab to try it out. You can manually type in a body of text or load it from a file or URL. Matching patterns in the body of text are automatically highlighted. From the Debug tab, you can examine how the parser evaluates the body of text against the regular expression.
RegexBuddy can also serve as a repository for your regular expressions. There is a Library tab where you can save your regular expressions. The Library also contains a variety of regular expressions for performing common tasks such as HTML and XML processing, and numeric and date string processing. RegexBuddy will even write the Visual Basic or C# code needed to use regular expressions from within your applications. Choose the language and action to perform—finding regular expression matches in a body of text, replacing matched regular expressions with another string, and so on—and RegexBuddy displays the corresponding source code.
Price: $39.95.
Regexbuddy.com
RegexBuddy Shows Regular Expressions as Text and as a Tree of Tokens

The Bookshelf
In The Productive Programmer (O'Reilly, 2008), Neal Ford shares proven techniques that will help any developer improve his or her productivity. The first part of the book explores behaviors and tools for boosting developer productivity; the second part looks at software development practices that help contribute to a more streamlined development process.
Early on, Ford focuses on the mechanics of using a computer productively. He provides tips and tools to decrease the amount of time required to perform common programming-related tasks. He espouses the Don't Repeat Yourself tenet for improving productivity and provides examples of how to avoid needless repetition in areas from version control to technical documentation.
Poor software development practices can quickly swallow up those productivity gains, Neal notes. Adding unnecessary features, insufficiently testing your code, and failing to correctly encapsulate your objects are all practices that lead to bugs and unmaintainable code.
The willingness to question the status quo is another important aspect in developer productivity. Too often developers get into a rut and use a particular design pattern or coding technique because that's the way it's always been done, overlooking or turning down alternatives that may be more efficient.
The Productive Programmer contains numerous code examples and discussions on particular frameworks that, unfortunately for .NET developers, focus on Java and Ruby. And although some of the tools mentioned are for UNIX-based or Mac OS systems, the majority of the ideas presented are not architecture, framework, or programming language dependent and will help any programmer build software applications better and faster.
Price: $39.99.
oreilly.com

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
International Applications
Melitta Andersen

This column is based on a prerelease version of Silverlight. All information is subject to change.
Contents
Localization with the CLR
Working with Text
Silverlight Globalization
Further Reading
Most programs present some sort of data to users. Clearly applications with a GUI, such as a Windows Presentation Foundation (WPF) or Windows Forms application, have been designed to provide information to users. However, even simple console applications display text. The process of globalization is making sure that your application can handle data that comes from different cultures. This can be as simple as making sure your calendar title is wide enough to handle Hawaiian month names or using the negative number format that your users expect.
Globalization is different from localization, which is actually translating resources from your application into different languages. Your application needs to be aware of globalization even if it is only in one language.
Here's an example of how there can be variations, even in an application that has not been translated. Certain types of data, such as dates, can be formatted in a variety of ways. Figure 1 lists just a few formats that use English names for months and days of the week. All the entries represent the same piece of data. But users have certain expectations about date formats, and they can indicate their preference via settings in the control panel. So if your program makes a call to DateTime.Now.ToString, any number of strings could be returned. If you have not specified a format, ToString automatically picks up the user's preferences.
Figure 1 Some Date Formats
The date string "1/2/03" is meaningless unless you know what each part of the string represents. You would think that knowing the format string would help you interpret the meaning of the data. But even that is no guarantee. Not everyone uses the same calendar. In fact, the System.Globalization namespace currently supports 15 different calendars. So the string "6/26/1980" interpreted as "M/d/yy" might represent different days in the Gregorian calendar and the Julian calendar. (There are a number of Web sites that can convert dates to different calendars.)
Interestingly enough, the same interpretation issues apply to numbers. Some users might expect the decimal separator to be a period, while others are used to seeing a comma. So when one user expects 1.23, another might expect 1,23. It gets even more complicated when you're dealing with negative numbers. If you look at System.Globalization.NumberFormatInfo.NumberNegativePattern, you will see that the Microsoft .NET Framework stores 5 patterns for the placement of the negative sign. And that's with only one symbol. It stores 16 different patterns for negative currency values in NumberFormatInfo.CurrencyNegativePattern.
As you might guess, when storing data as strings you should always explicitly indicate the format you wish to follow and use that same format for both the ToString and Parse methods. Remember, some strings can be ambiguous without context. Of course, all this fancy formatting matters only for display purposes; it is a best practice to pick an invariant representation for stored data that will only be read by a machine. The Framework provides a mechanism for this with CultureInfo.InvariantCulture.
With so many options for formatting, how are you to know which one to choose? Fortunately, the .NET Framework has an infrastructure in place for formatting data for display. As you've seen, user preferences can be followed automatically. But how does the Framework decide what to use when a user hasn't specified?
Taking a closer look at the System.Globalization namespace can help answer this question. The contents of System.Globalization are used to describe information about a culture. We already know that it contains 15 calendars and 16 ways to format negative currency values. In addition, it has other culture-specific properties for time and numbers, as well as classes for storing information about languages and regions. It also contains a mechanism to organize and group this information for use. That mechanism is the CultureInfo class, which stores information about a culture, including number formatting, date formatting, and calendar, as well as names for the culture and information about the writing system.
CultureInfo objects are created using an identifier for a particular culture. For example, in order to create an object that contains information about Swedish as it is used in Finland, you would specify the name "sv-FI" in the constructor. The Framework includes information on many cultures, and Windows also has data for a variety of locales. In addition to this, users can create their own custom culture.
CultureInfo implements IFormatProvider, which is a type that ToString can accept as an argument for dates and numbers. So to go back to the original DateTime.Now.ToString example, if you wanted to get the date formatted with the default format associated with the sv-FI culture, you could write the following:
DateTime.Now.ToString(new CultureInfo("sv-FI"));
If there were no user overrides for sv-FI and the date in question happened to be "6/26/1980" again, the date would print as "26.6.1980" followed by the time string. If you wanted to get the long date format associated with sv-FI, you could write:
CultureInfo ci = new CultureInfo("sv-FI");
Console.WriteLine(DateTime.Now.ToString(ci.DateTimeFormat.LongDatePattern, ci));
This would yield the string "den 26 juni 1980." Note that there the LongDatePattern, which only includes the date portion of the DateTimeand omits the time. These examples work because the formatting information stored in the CultureInfo object is used.
That explains how ToString works when specific formatting information is passed in. But as you saw at the very beginning, there are overloads of ToString that take no formatting information at all. Yet the data is still formatted. Where does the information come from?
One of the static properties on the CultureInfo class is CurrentCulture. This returns the culture that is associated with the current thread. This property can be set via Thread.CurrentThread.CurrentCulture. When threads are created, they default to the current culture of the operating system. If no format information is specified for a ToString or Parsing operation, the information in CultureInfo.CurrentCulture is used.
Culture information in general is subject to change. A country might adopt the Euro or change time zone or add another date format. So even if you specify a culture, the data might change between uses. Remember, the globalization information is for display only. If you need to store data as strings, the InvariantCulture property on CultureInfo is a good choice.

Localization with the CLR
What if you want to localize your application to more than one language? Doing this might involve all of the data display functionality handled in System.Globalization, as well as the translation of different parts of your application. You might also want to display different images for different cultures. One way that you could go about doing this would be to make multiple copies of your application, one for each language to which you want to localize your product.
This is quite inefficient, though. Fortunately, the Framework provides a way to abstract these localizable resources via classes found in the System.Resources namespace. The main classes for creating, reading, and using resources are ResourceWriter, ResourceReader, and ResourceManager.
The ResourceWriter class allows you to store resources as pairs of names and values. When you create an instance of ResourceWriter, it is for a particular file. The naming convention for these files is [basename].[culturename].resources. "Basename" is used for organizing your resources and is the name of an application or a class. You can use the CultureInfo.Name property to find the culture name.
U.S. English resources for the sample application MySampleApplication, for example, would be put in the file "MySampleApplication.en-US.resources." In addition, apps should have a default resource file for the neutral culture named [basename].resources. It is generally a good idea to package your resource files in satellite assemblies so that you can version the localization data separately from the main assembly. However, it's always a good idea to include one set of resources, the neutral resources, in the main assembly so that resource lookup will always have at least one set of resources to find.
The ResourceReader class allows you to enumerate the name/value pairs from a resource file. However, the most commonly performed task is to look up a particular resource for a particular culture. This is accomplished via the ResourceManager class. You can access individual resources with the GetObject and GetString methods, or you can load all the resources for a particular culture into a Hashtable using the GetResourceSet method.
Just like the globalization information, resources are stored by culture. And just as globalization can be configured with a default culture, the ResourceManager uses CurrentCulture.CurrentUICulture for its default culture. Again, this is a property of the current thread. CurrentCulture and CurrentUICulture often match, but they regulate different things. CurrentCulture controls the formatting of data, while CurrentUICulture determines which resources are loaded.
The framework carries around most of the data needed by classes in System.Globalization, but an application controls its own resources. If an application does not localize to a particular culture, the resources will not be available in that culture. But in general, you do not want resource lookup to fail. That is why you include the neutral resources in the main assembly in the first place—so that the application always has a set of resources to fall back to.
One property of CultureInfo that hasn't been mentioned is the Parent property. Culture information is arranged in a hierarchy where a neutral culture, that is a culture not associated with a particular region, is the parent of a culture associated with a particular region. For example, the parent of en-US, en-CA, and en-UK is the neutral culture en. If a resource for a culture is not found, the ResourceManager will probe the chain of parents until one is found. This will end with the InvariantCulture, the resources embedded into the main assembly.
The hierarchy of cultures allows you to factor common resources out of specific cultures and into neutral cultures. Let's say as an example that I want to localize my application into en-US, en-CA, and en-UK. Many of the resources that are needed for these cultures are likely the same with only a few differing. In this case, I can store all of the common resources in a file for the neutral culture en, putting only the differences in the resource files for the specific cultures.
If you had the resources that are illustrated in Figure 2 and tried to look up the resource corresponding to the name "greeting," en-US would return "hi" while en-UK and en-CA would both fall back to the en resource "hello."
Figure 2 Resource Hierarchy

Working with Text
Text introduces a whole new set of complications. For instance, how do you actually represent all of the characters you might need for translating your text into other languages? Strings are represented in the .NET Framework by the System.String class. Fortunately, strings in the CLR use Unicode, specifically the UTF-16 encoding, so they can represent standard characters.
The System.Text namespace includes some classes that can be used for encoding Unicode characters into bytes and decoding bytes into characters. This allows you to translate between different Unicode encodings if you need to.
While Unicode is the current standard, there are some previous encodings that represented only characters used in a particular language or region. These encodings are known as "code pages." If you need to work with text encoded with one of these old code pages, the System.Text classes will allow you to work with those as well. In general, this is only necessary for working with legacy applications. New applications should use the Unicode standard.
Some strings might just be printed on the console or in a label, but often you will want to perform some action with a string. Perhaps a list of strings would be easier to read if it were sorted in the display. Maybe you want to convert characters in the string to a different case. These actions may seem simple enough, but looks can be deceiving. There is a fundamental problem in word sorting that might not immediately be apparent. In English, for example, there is a 26-character alphabet with a fixed ordering. But the characters have multiple cases. How do you sort instances of the same word with different casing?
The Framework has three modes of string comparison: ordinal, word, and string. The ordinal comparison looks at the numeric value of each character to compare, which means that ordering based on String.CompareOrdinal would consider "alphabet" greater than "Alphabet." Word comparison is the default. It is culture sensitive and might treat certain non-alphanumeric characters, such as the hyphen, as a special case. It can give them a small weight or consideration so that, for example, "a-lphabet" and "alphabet" sort near each other. String comparison is also culture sensitive but has no special cases, and the non-alphanumeric characters all sort before the alphanumeric ones.
If you were to do a culture-sensitive comparison for en-US using String.Compare, you would see that "Alphabet" is greater than "alphabet." If you were to do an ordinal comparison ignoring case, they would be equal. System.Globalization.CompareOptions includes nine different options that indicate which parameters to take into account when sorting.
Each culture has its own set of characters and sort order. Some have more than one. For example, Spanish (Spain), es-ES, uses International sorting by default, but can use the Traditional sort order as an alternative. Chinese (China), zh-CN, sorts by pronunciation by default, but can also sort by stroke count. Each CultureInfo object in the Framework has a CompareInfo property that is used for comparing strings. The Framework stores this data.
Generally speaking, the culture-sensitive ordering is much better for sorting and displaying. But when it comes to testing equality of strings, especially in certain cases when there might be security implications, ordinal comparison is definitely the best choice. Ordinal comparison looks only at the value of the compared characters; therefore, it is consistent while culture-sensitive comparisons may have different results based on the culture that is used.
One of the most commonly cited problems of using culture-sensitive string comparisons for security purposes is known as "the Turkish I problem." In most Latin alphabets, the character i (\u0069) is simply the lowercase version of the character I (\u0049). Most people using those alphabets have no idea that there could be any variation in this, so it's just considered to be a default. However, in Turkish (tr-TR) that mapping is incorrect. The uppercase version of the character i is the character İ (\u0130), or an uppercase I with a dot. Similarly, the lowercase version of the character I is the character ı (\u0131), or a lowercase i without a dot.
Consider the case where you want to check to see if a URI (Uniform Resource Identifier) starts with the string "FILE:". You want to do this in a case-insensitive way to make sure words such as "file:" don't get past your filter. If you compare "file:" to "FILE:" using the en-US culture and ignoring case, they will be equal. But if the tr-TR culture is used, they will not be equal. String.Compare defaults to the CurrentCulture.
So for such situations, you should use StringComparison.OrdinalIgnoreCase. An even better option is to use String.Equals, which is ordinal by default, if you are just testing equality. Figure 3 gives you an idea of the results of ToUpper and ToLower for the letter I. For more information on comparing strings, please see "New Recommendations for Using Strings in Microsoft .NET 2.0".
Figure 3 Results of ToUpper and ToLower

Silverlight Globalization
Globalization is especially important to Silverlight applications. With a desktop application, you can know exactly who your target audience is. But a Silverlight application on the Web can be seen by anyone. Silverlight has much of the same globalization infrastructure as the desktop Framework, but it gets most of its data from the underlying operating system. This helps a Silverlight application running on Windows look more like a Windows-based application and a Silverlight application running on a Mac OS look more like a Mac OS application.
Many of the differences between the desktop CLR and Silverlight (CoreCLR) stem from the fact that for speed and ease of download, a smaller runtime was needed for Silverlight. As such, Silverlight gets much more information from the underlying operating system rather than from carrying it all around in the CoreCLR. For instance, the CoreCLR doesn't have the sorting tables stored, so it only has access to those that the operating system has. Additionally, Silverlight only uses Unicode and does not have the legacy code pages. It also gets culture information from the operating system. This means that you really have to be prepared to handle a wide variety of data.
It has always been the case that data might not be available if it relied on a custom culture or an OS culture. But with all of the data coming from the OS, a single Silverlight application is more likely to run into different data. It is a good idea to make sure your target OS supports the UI cultures you want to use and to make sure your application behaves in a reasonable manner when the requested culture is not present.
Culture data is baked into the .NET Framework, making it difficult to update at the same frequency at which the data changes. This leads to data that can be out of date. It can also lead to the situation in which native applications and managed applications display different data. In Silverlight, you get consistency between the two, and applications can take advantage of updated OS information.
Using OS data also allows you to take advantage of more information. For example, in the desktop version of the .NET Framework, every CultureInfo has a single-parent culture, and to look up resources, all the ResourceManager has to do is simply walk the parent chain.
Take an application with de-DE resources and fr-FR resources. Now let's say that you have a user with es-ES as the primary culture but who also knows German and could read de-DE resources. The ResourceManager would fall back to the invariant culture even though resources existed that the user could understand. Most modern operating systems allow users to specify multiple language preferences. In Silverlight, we modified the ResourceManager to follow those preferences before falling back to the invariant culture.
Silverlight also contains a few differences in string comparison that are related to security rather than size. The default for most of the operations that involve string comparison in the desktop Framework is CurrentCulture. In Silverlight, the operations that involve partial matching, which includes both String.IndexOf and String.StartsWith, now use ordinal comparison by default. Compare and CompareTo still use CurrentCulture for the default, though you should really think about whether or not you are using the correct overload for the specific behavior you want. ToUpper and ToLower now use the InvariantCulture by default. Equality still uses ordinal comparison.

Further Reading
If you would like to have more information about building world-ready applications for Silverlight and Windows, a couple of good resources for you to look include the top-level MSDN pages for System.Globalization, for System.Resources, and for System.Text . You can find links to more detailed information there, as well as helpful example code. Finally, the Go Global Developer Center, is also an excellent resource for information about building international applications.

Send your questions and comments to clrinout@microsoft.com.

Melitta Andersen is a Program Manager on the Base Class Libraries team of the CLR. She mainly works on numerics, collections, and some of the infrastructure detailed in this column.


Data Points
Cloud Gazing From Silverlight 2
John Papa

Code download available at: DataPoints2008_11.exe (151 KB)
Browse the Code Online
This column is based on a prerelease version of Silverlight 2. All information is subject to change.
Contents
How are Web services handled in Silverlight 2?
How is data passed from services to Silverlight 2?
How is data consumed from ASMX Web services?
How is data consumed from WCF Web services?
How is data received from a REST Web service?
How do I query XML?
Do I need to worry about cross-domain policies when I'm accessing third-party Web services?
Developers who use Silverlight have their heads in the clouds, but can you blame them? Obtaining data for rich Internet applications (RIA) from various types of Web services is growing in prominence. Silverlight apps can communicate with ASMX Web services, Windows Communication Foundation (WCF) Web services, Representational State Transfer (REST) services, and plain old XML (POX) services. Whether these services are supplied by third parties or they are custom services hosted on the same server where the Silverlight app resides, Silverlight can request data, consume the data and pass data back and forth between the Silverlight client application and these types of Web services.
Based on questions I have been receiving, it's clear that developers want to know more about calling services from Silverlight 2 applications, the differences between using WCF, ASMX, and REST services, and about how to consume the data that these services expose. In this installment of Data Points, I will address some of these questions and demonstrate how to deal with data services.
There are plenty of third-party Web services that offer everything from access to their APIs to storage of client-specific and custom data. I'll discuss the consuming and passing of data between Silverlight applications and the cloud of services. I'll also show how data can be consumed using several XML parsing libraries. For example, XML documents can be opened and iterated through or XML can be queried using LINQ to XML. Services such as Flickr, Amazon, Twitter, and Live Search all expose APIs that can be communicated with through various Web service techniques. Here I'll discuss some of these APIs and show how you can communicate with them using REST- and SOAP-based services. All examples in this column are available in both C# and Visual Basic and can be downloaded from the MSDN Magazine Web site.

How are Web services handled in Silverlight 2?
This is a good question I like to start with, as it sets the basis for understanding why so many developers are using Silverlight 2 to interact with Web services. Silverlight 1.x did not support code based on the Microsoft .NET Framework, nor were there any .NET controls. Silverlight 2 introduced a lot of full-featured functionality that blew away previous limitations. Silverlight 2 allows you to write code in C# or Visual Basic and leverage all your existing experience against the .NET CLR. While the libraries included in Silverlight are a subset of the overall .NET libraries, there is plenty of functionality in Silverlight 2. For example, the WebClient and HttpWebRequest classes are accessible from Silverlight and can be used to interact with Web-based services by calling URIs. The data can then be consumed using XmlReader objects or through LINQ to XML.
Silverlight 2 added a large set of features for passing data between services. The following features are all new to Silverlight 2:
  • SOAP-based Web services can be accessed through proxy classes.
  • REST-based Web service can be accessed.
  • ADO.NET Data Services, a REST-based service that allows remote LINQ queries, can be accessed.
  • Results can be retrieved using both JavaScript Object Notation (JSON) and XML from Web services.
  • Duplex communications are supported through WCF (using server push).
  • Cross-domain access is supported using either the clientaccesspolicy.xml file or the crossdomain.xml file.
  • Cross-domain network support is available using HTTP and Sockets.
  • Web service calls are initiated asynchronously.

How is data passed from services to Silverlight 2?
Figure 1 shows several types of services that Silverlight 2 can access. Data can be passed between Silverlight 2 and these services as XML, JSON, or as scalar values. SOAP-based services allow themselves to be described, which, in turn, allows the data that they expose to be described. A Silverlight 2 client can access the data from SOAP-based services by creating a proxy to the service, which generates a class description for the exposed classes.
Figure 1 Accessing Services from Silverlight
SOAP-based services describe themselves so the Silverlight 2 client application can pass data to and from the services using any exposed entity. Both ASMX Web services and WCF Web services allow entities to be included as part of their contract with a client reference. The client reference generates a proxy class on the client that contains a definition for the exposed classes and the service methods (all service methods are converted to asynchronous calls for Silverlight 2 clients).
Services that do not describe themselves, such as POX and REST-based services, allow a client application to call their service methods and retrieve data in scalar format, as XML, or as JSON. These types of services do not expose Web Services Description Language (WSDL) and therefore the client application does not generate a proxy class for them. The services are queried using a URI through classes like the WebClient or the HttpWebRequest classes.

How is data consumed from ASMX Web services?
Like ASMX services, SOAP-based WCF Web services describe themselves through WSDL. If the client adds a reference to the service, it will generate a client proxy to the service, allowing the client to pass data with the service in their native form (that is, as classes). For example, a Dog entity may be returned from a SOAP-based Web service (ASMX or WCF) to a Silverlight 2 client. The Silverlight 2 client can then create an instance of the Dog class. Methods are exposed to clients in ASMX Web services by decorating the public service methods with the WebMethod attribute and decorating the ASMX Web service class with the WebService attribute. Figure 2 illustrates this in C#.
Figure 2 Returning an Entity from a Web Service in C#
[WebService(Namespace = "http://www.microsoft.com/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.ComponentModel.ToolboxItem(false)]
public class TestService : System.Web.Services.WebService {
  [WebMethod]
  public List<Dog> GetDogList() {
   return new List<Dog> {new Dog {Name = "Spot", Age = 10},      new Dog {Name = "Kadi", Age = 13}};
  }
}

[Serializable]
public class Dog {
  public string Name { get; set; }
  public int Age { get; set; }
}
This example shows the .NET-targeted code in a Silverlight 2 client app that generates the proxy to the SOAP service. An event handler is assigned to the GetDogListCompleted event followed by the asynchronous call to the service. All calls to Web services from Silverlight 2 are made asynchronously to prevent the services from tying up the UI thread and freezing the user. Here's the event handler receiving the results as an array of the entity type:
TestServiceSoapClient proxy = new TestServiceSoapClient();
proxy.GetDogListCompleted +=   new EventHandler<GetDogListCompletedEventArgs>(proxy_  GetDogListCompleted);
proxy.GetDogListAsync();
Here's the C# code for receiving the entities:
void proxy_GetDogListCompleted(object sender,
  GetDogListCompletedEventArgs e)
{
  Dog[] list = e.Result;
}

How is data consumed from WCF Web services?
Like ASMX Web services, WCF Web services describe themselves through WSDL. WCF Web services expose their services and the data that they can pass using a series of attributes. This means that the client applications can reference a WCF Web service and generate a proxy class to interact with the services and their exposed data contracts.
WCF defines a data contract as any custom .NET type that can be returned from a WCF Web service. These types are decorated with the DataContract attribute. Properties of the class must be decorated with the DataMember attribute. Figure 3 shows a sample class decorated with the appropriate attributes.
Figure 3 Preparing Data for WCF
[DataContract]
public class Employee {
  int _employeeID;
  string _firstName;
  string _lastName;
  string _title;
  DateTime _hireDate;
  byte[] _photo;

  [DataMember]
  public int EmployeeID {
    get { return _employeeID; }
    set { _employeeID = value; }
  }
  [DataMember]
  public string FirstName {
    get { return _firstName; }
    set {_firstName = value; }
  }
  [DataMember]
  public string LastName {
    get { return _lastName; }
    set { _lastName = value; }
  }
}
Once the data formats have been defined for the WCF Web service using the DataContract and DataMember attributes, the service and its service methods must be set up. The services are decorated with the ServiceContract attribute and its methods are decorated with the OperationContract attributes. If desired, the service can implement an interface, in which case the interface (and not the service class) and its interface methods are decorated with the attributes. Here is the IEmployeeService and its members decorated with these attributes:
[ServiceContract(Namespace = "")]
   public interface IEmployeeService
   {
    [OperationContract]
    List<Employee> FindEmployees();
   }
Note here that Silverlight 2 applications can only consume SOAP-based WCF services that use basicHttpBinding.
A Silverlight 2 client application can add a service reference to this service, which creates a proxy class. The service can then be called asynchronously, just like with an ASMX Web service, and a handler can help capture the results. The code to implement the proxy and the completed event handler use exactly the same format, as shown earlier for ASMX Web services. Once the data is returned as a List<Employee> the entities can be iterated, data bound, modified, and passed back through a service method if required.

How is data received from a REST Web service?
REST services can be called through a URI using a query string with parameter. It can be called from Silverlight 2 using the WebClient class or the HttpWebRequest. Making calls through WebClient is simpler, but HttpWebRequest offers more control over how the requests are made. Also, WebClient returns on the UI thread while HttWebRequest returns on the background thread. HttpWebRequest's callback must use the Dispatcher to interact with the UI. The following code shows the System.Net.WebClient retrieving recent stories from the Digg service:
string baseUri = "http://services.digg.com/stories/topic";
string topic = txtTopic.Text;
string appKey = "http%3A%2F%2Fwww.microsoft.com";
int count = int.Parse(txtTopicCount.Text);
string url = String.Format("{0}/{1}?appkey={2}&count={3}", baseUri, topic, appKey, count);
WebClient svc = new WebClient();
svc.DownloadStringAsync(new Uri(url));
Here the data is returned from the WebClient call as raw XML which can then be parsed or queried. If the REST service has parameters, they can be passed as part of the querystring.

How do I query XML?
When a REST-based or POX service is invoked, it can return data as XML (REST services can also return data as JSON). The XML can then be parsed using a number of XML parsing libraries. However, the XML can also be queried using LINQ to XML, which offers a rich query interface to extract data from XML structures without having to resort to iterating through complex XML hierarchies.
The REST service in the example you just saw returns raw XML from the Digg service that shows the most recent stories. This XML can then be parsed using an XML library or it can be queried using LINQ to XML, as shown in Figure 4. The XDocument class's Parse method can consume the XML from the REST service shown earlier. The XML can then be queried using LINQ to XML. The code shown in Figure 4 demonstrates the LINQ to XML syntax for querying the XML shown in Figure 5.
Figure 4 LINQ to XML Querying Digg Data
XDocument xml = XDocument.Parse(rawXml);
var storiesQuery = from story in xml.Descendants("story")
  select new DiggStory
    {
       Id = (int)story.Attribute("id"),
       Title = ((string)story.Element("title")).Trim(),
       Description = ((string)story.Element("description")).Trim(),
       ThumbNail =
         (story.Element("thumbnail") == null
              ? string.Empty
              : story.Element("thumbnail").Attribute("src").Value),
       Link = new Uri((string)story.Attribute("link")),
       DiggCount = (int)story.Attribute("diggs")
    };
Figure 5 Digg XML Response
<?xml version="1.0" encoding="utf-8" ?>
<stories timestamp="1222547017" min_date="1219955010" total="2541"
                                                 offset="0" count="10">
  <story link="http://news.msn.com.us/2/hi/technology/7540282.stm"
       submit_date="1222545893" diggs="1" id="8725656" comments="0"
       href="http://digg.com/microsoft/Microsoft_releases_SQL_Server_
       2005_recently" status="upcoming" media="news">
    <description>Microsoft has released SQL Server 2005 recently
       as an upgrade to the popular SQL Server database.</description>
    <title>Microsoft releases SQL Server 2005</title>
    <user name="D3a1i0" icon="http://digg.com/img/udl.png"
      registered="1104422772" profileviews="5616" />
    <topic name="Microsoft" short_name="microsoft" />
    <container name="Technology" short_name="technology" />
    <thumbnail originalwidth="226" originalheight="170" contentType=
      "image/jpeg" src="http://digg.com/microsoft/Microsoft_releases_SQL_
      Server_2005_recently/t.jpg" width="80" height="80" />
  </story>
  <story link="http://navigatetrends.blogspot.com/2008/09/microsofts-
      virtual-receptionist.html" submit_date="1222545326" diggs="1"
      id="8725560" comments="0" href="http://digg.com/microsoft/
      Microsoft_s_virtual_receptionist" status="upcoming" media="videos">
    <description>Microsoft's virtual receptionist</description>
    <title>Microsoft's virtual receptionist</title>
    <user name="billarunk" icon="http://digg.com/img/udl.png"
      registered="1216563445" profileviews="341" fullname="bill.arunk"
    />
    <topic name="Microsoft" short_name="microsoft" />
    <container name="Technology" short_name="technology" />
    <thumbnail originalwidth="400" originalheight="254" contentType="image/
      jpeg" src="http://digg.com/microsoft/Microsoft_s_virtual_
      receptionist/t.jpg" width="80" height="80" />
  </story>
...
...
</stories>
The query starts in the stories/story hierarchy and queries all of the story elements. There is no namespace specified for this XML; however, I left the namespace variable (ns) in place to show how to prefix all XML paths in case the results do refer to a namespace. Driving story elements is accomplished using the XDocument instance's Descendants method. The story element's properties can then be retrieved using the story variable and its Element method. For example, the following line of code grabs the value of the title element from the story element and sets it into a property named Title:
Title = ((string)story.Element("title")).Trim()
The LINQ to XML query can return the results through a projection or into a class structure. The code shown in Figure 4 creates a DiggStory class to store the results of each status element retrieved from the XML data. The DiggStory class that the developer has defined, where all of the properties (for simplicity) have been defined as string. LINQ to XML could include other aspects such as ordering or filtering criteria, too, if needed.
Using LINQ to XML to query and parse XML data from REST or POX services allows developers to leverage their existing knowledge of other LINQ forms since the syntax is very similar across them all. Perhaps the biggest benefit of using LINQ to XML to parse the data is the power and simplicity of querying the data without having to resort to iterating through elements in nested foreach loops.

Do I need to worry about cross-domain policies when I'm accessing third-party Web services?
When a Silverlight 2 application makes a Web service call, the call is precluded by a check on the Web service's server for a cross-domain policy file. This check is done when the Silverlight application is hosted on a different domain than the Web service. For example, Figure 6 shows that a Silverlight 2 application hosted on the Web server johnpapa.net can call a Web service also on johnpapa.net since they are on the same domain. However if the Silverlight 2 client hosted from johnpapa.net wants to call a Web service hosted on microsoft.com, Silverlight 2 checks to make sure the cross-domain policy file on microsoft.com exists and allows the call.
Figure 6 Cross-Domain Checking
For more information on Cross-Domain Policies, see my September 2008 column, "Service-Driven Apps With Silverlight 2 And WCF". In that column, I discuss the file formats and how the policies work.
Whether the call is made to a WCF Web service, a REST-based service or any type of Web service, Silverlight 2 only allows cross-domain calls to occur if the target Web server allows the call through its cross-domain policy file. For example, the code you saw earlier
string baseUri = "http://services.digg.com/stories/topic";
string topic = txtTopic.Text;
string appKey = "http%3A%2F%2Fwww.microsoft.com";
int count = int.Parse(txtTopicCount.Text);
string url = String.Format("{0}/{1}?appkey={2}&count={3}", baseUri,
  topic, appKey, count);
WebClient svc = new WebClient();
svc.DownloadStringAsync(new Uri(url));
invokes the Digg Web services to access the most recent stories. Digg includes a cross-domain policy file using the Flash format at the location http://services.digg.com/crossdomain.xml.
The contents of this Digg cross-domain file at the time of this writing are shown here:
<cross-domain-policy>
  <allow-access-from domain="*"/>
</cross-domain-policy>
Notice that the Digg cross-domain file allows calls from any domain. However, this could be restricted, just like the Twitter cross-domain file enforces restrictions. Notice the Twitter cross-domain file in the following:
<?xml version="1.0" encoding="UTF-8" ?>
<cross-domain-policy
  xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xsi:
  noNamespaceSchemaLocation="http://www.adobe.com/xml/schemas/
  PolicyFile.xsd">
  <allow-access-from domain="*.twitter.com" />
  <site-control permitted-cross-domain-policies="master-only" />
  <allow-http-request-headers-from domain="*.twitter.com" />
</cross-domain-policy>
The cross-domain file for Twitter restricts calls to those that originate from the *.twitter.com domain. Your Silverlight 2 application likely is not hosted on the twitter.com domain, so a Silverlight client cannot make a Web service request to Twitter's REST-based services directly. This is where a proxy service can help you out. For example, you could create a WCF Web service hosted on your domain that calls the Twitter.com Web services. The calls from your WCF services relay the information between the Silverlight application and Twitter. The WCF service running on the server does not suffer the cross-domain policy file restrictions as a Silverlight client does, so it can make the calls. Another option is to relay the calls through a service such as Popfly or Yahoo's cloud services.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa (johnpapa.net) is a Senior Consultant with ASPSOFT (aspsoft.com) and a baseball fan who spends summer nights rooting for the Yankees with his family. John, a C# MVP and INETA speaker, has authored several books and is now working on his latest, Data-Driven Services with Silverlight 2. He often speaks at conferences such as DevConnections and VSLive.


Cutting Edge
Browser Interoperability In Silverlight 2
Dino Esposito

This column is based on a prerelease version of Silverlight 2. All information is subject to change.
Code download available at: CuttingEdge2008_11.exe (565 KB)
Browse the Code Online
Contents
Configuring the Silverlight Control
Accessing the DOM from Silverlight
Inside the Browser Interoperability Layer
Attaching Managed Code to DOM Events
Synchronous Calls and Silverlight 2
XAML, Managed Code, and JavaScript
Cross-Domain Access and Silverlight Plug-Ins
In a Nutshell
You can use Silverlight 2 to build full-page Windows Presentation Foundation (WPF)-like Web applications or to enrich HTML-based pages with additional features such as animations, advertising, and specific applets. The browser doesn't directly process the eXtensible Application Markup Language (XAML) content that makes up a Silverlight application. Instead, within the HTML page, an <object> tag points to the Silverlight 2 plug-in and has among its parameters the URL to download any necessary XAML resources.
The bottom line is that there's always an HTML page wrapped around a Silverlight plug-in, even though you often can't see it because the Silverlight application is configured to run in full-screen mode. In addition, the content hosted by the Silverlight plug-in is not isolated from the surrounding page in a same-site situation. When the Silverlight content comes from a different domain than the hosting HTML page, then it is indeed isolated from the page.
Silverlight comes with a browser interoperability layer that allows managed code to access the document object model (DOM) of the underlying page and register managed handlers for page-level events. At the same time, any JavaScript code running in the page can gain access to the XAML content of the plug-in and even make modifications. Finally, JavaScript code running in the page can also invoke managed functions as long as these functions are exposed properly. This month I'll discuss the browser interoperability layer of Silverlight 2 and the ways you can take advantage of it in your applications.

Configuring the Silverlight Control
As mentioned, the Silverlight plug-in is invoked through an <object> tag. However, this tag can be created for you by an ASP.NET control in ASP.NET pages. When you create a sample Silverlight application in Visual Studio 2008, the following markup for the Silverlight server control is automatically inserted in the test page:
<asp:Silverlight ID="Xaml1" runat="server"
     Source="~/ClientBin/SilverTestApp.xap"
     MinimumVersion="2.0.30523"
     Width="100%"
     Height="100%" />
The MinimumVersion attribute indicates the minimum version of the Silverlight runtime that is required to run the application specified in the Source attribute. Width and Height define the size of the Silverlight window with respect to the host page. As you can see, by default the Silverlight plug-in is configured to run in full-size mode.
Figure 1 lists the properties you can set to configure the Silverlight plug-in. The properties in Figure 1 refer to the Silverlight ASP.NET control that executes on the server side and spits out an <object> tag that may have differently named parameters or no equivalent parameters. For the purposes of this column, you should pay a lot of attention to the HtmlAccess property. (Note that this part of the Silverlight SDK changed a bit in the transition from Beta 1 to Beta 2.)
Figure 1 Properties of the Silverlight ASP.NET Server Control
Property | Description
AutoUpgrade | Indicates whether the Silverlight plug-in should automatically be upgraded. The default is false.
DefaultScriptType | The default type of the client JavaScript object to create and associate with the Silverlight plug-in. By default, it is Sys.UI.Silverlight.Control.
EnableFrameRateCounter | Indicates whether to display the current frame rate in the hosting browser's status bar. The default is true.
EnableRedrawRegions | Indicates whether to show the areas of the Silverlight plug-in that are being redrawn for each frame. The default is true.
HtmlAccess | Indicates whether to allow the Silverlight application to access the page DOM.
InitParameters | Defines an optional set of user-defined initialization parameters.
MaxFrameRate | Maximum number of frames to render per second for the Silverlight document.
MinimumVersion | Minimum version of the plug-in that is required for the current application. The default is 1.0.
PluginBackground | Indicates the plug-in background color.
PluginNotInstalledTemplate | HTML markup to render if the Silverlight plug-in is not installed.
ScaleMode | Indicates how the Silverlight plug-in fills the available space: none, zoom, or stretch. The default is none.
ScriptType | The type of the client JavaScript object to create and associate with the Silverlight plug-in.
Source | The URL for the XAML source file or the Extensible Ajax Platform (XAP) source package to download.
SplashScreenSource | The URL of the splash screen document to render when the source document is loading.
Windowless | Indicates whether the Silverlight plug-in is rendered directly in the browser's client area or within an aptly created window. The default is true.
The HtmlAccess property indicates the level of access to the underlying page DOM that the Silverlight application is allowed. The property accepts values from the HtmlAccess enumerated type. Values are listed in Figure 2.
Figure 2 Values to Control Access to the Page DOM
Value | Description
Disabled | The application is not allowed access to the DOM of the underlying page.
Enabled | The application is allowed access to the DOM of the underlying page.
SameDomain | The application is allowed access to the DOM of the underlying page only if the page comes from the same domain of the Silverlight XAP application. This is the default setting.
Obviously, Enabled and Disabled are self-explanatory. The default setting is SameDomain, which doesn't inject any script in the page's markup. Note that a Silverlight application can be hosted in a page that is, in turn, hosted within a frame outside of its native domain. In this case, the Silverlight managed code is able to access the DOM of the host page in a cross-domain manner. Browsers have their own barriers to prevent cross-domain scripting, but they can't do much to stop the managed code within the Silverlight plug-in. Authors of Silverlight pages use HtmlAccess to control cross-domain access.

Accessing the DOM from Silverlight
Once access to the underlying page DOM has been granted, a Silverlight application can use the members on the static class HtmlPage to accomplish its own tasks. Figure 3 lists properties and methods of the HtmlPage class.
Figure 3 Members of the HtmlPage Class
Member | Description
BrowserInformation | Gets general information about the browser, such as name, version, and operating system.
Document | Provides access to the document object of the page.
IsEnabled | Indicates whether access to the page DOM is allowed.
Plugin | Gets a reference to the Silverlight object.
Window | Provides access to the window object of the page.
RegisterCreateableType | Registers a managed type as available for creation from JavaScript code.
RegisterScriptableObject | Registers a managed object as scriptable by JavaScript code in the page.
UnregisterCreateableType | Unregisters a type that was previously registered as createable using the RegisterCreteableType method.
The Document property returns a reference to an object of the managed type System.Windows.Browser.HtmlDocument, which represents the DOM of the underlying page. The Document property is strongly typed and exposes to Silverlight applications most of the features available to script in a weakly typed manner. Properties include the list of cookies, the query string, the body of the document, and the reference to the root of the DOM. The object also features a Boolean IsReady property to signal whether the browser's document has been fully loaded, which is paired with transposition of the DocumentReady event in the browser's DOM.
The HtmlDocument class also features methods such as Submit, AttachEvent, DetachEvent, GetElementById, and CreateElement. In addition, a couple of inherited methods—GetProperty and SetProperty—allow you to get and set attributes of the HTML elements from within managed code.
Full browser information is available through the BrowserInformation property. Also, in this case, the property is of a managed type that wraps any user agent information that is available at the browser level. The following code snippet shows how to access user-agent data:
string info = HtmlPage.BrowserInformation.UserAgent;
The BrowserInformation object also exposes a Boolean property that indicates whether the browser supports cookies.
The following code shows how to retrieve a reference to a DOM element using managed code:
HtmlElement label1 = HtmlPage.Document.GetElementById("Label1");
label1.SetProperty("innerHTML", "Dino");
The GetElementById method takes an ID string and attempts to locate a corresponding element in the underlying DOM. The object returned is of type HtmlElement—a managed type acting as the wrapper for a reference to an underlying browser object.
HtmlElement features a set of properties that makes it look like an HTML element. It has methods such as AppendChild, RemoveChild, GetAttribute, RemoveAttribute, and SetAttribute. Properties are Id, Parent, TagName, and Children.
All HTML script objects you manipulate from within Silverlight code derive from the base ScriptObject class. This class defines a couple of methods—SetProperty and GetProperty—that all derived classes such as HtmlDocument inherit.
What's the difference between getting and setting attributes and properties? Attributes are always managed as strings; properties are managed as strongly typed values.

Inside the Browser Interoperability Layer
A graphical view of the Silverlight browser interoperability layer and access to the page DOM is shown in Figure 4. Any requests made to any classes in the interoperability layer are resolved through an internal browser host service. Information is marshaled down to the browser's unmanaged environment and then back to Silverlight. Type differences are hidden and taken care of by the interoperability layer. DOM-level objects are wrapped as managed objects and served to the Silverlight code through a new managed interface—HtmlDocument, HtmlWindow, and the like. Let's focus on the GetElementById method.
Figure 4 The HTML Bridge in Silverlight 2
As its first step, the method ensures that the code is being called on the Silverlight UI thread. If not, an exception is thrown. Next, a request is made to the underlying browser to get a reference to the specified DOM element. If the request is successful, the method gets an unmanaged reference for the DOM object for which it will create and return a managed wrapper.

Attaching Managed Code to DOM Events
A very nice result of the Silverlight and DOM interaction is the ability you're given to run managed code in response to DOM events. For example, when the user clicks on a button, you can execute C# code instead of JavaScript code. Here's how you can achieve that:
HtmlElement button1;
button1 = HtmlPage.Document.GetElementById("Button1");
button1.AttachEvent("click",
         new System.EventHandler(Button1_Click));
You first retrieve a managed reference to the button (or the DOM element) of interest. Next, you invoke the managed AttachEvent method to register a handler for the specific event. The really nice thing is that the handler is managed code whereas the event is triggered at the browser unmanaged level. For example, getting a GUID is nearly impossible in JavaScript. It becomes fairly easy, however, if you can rely on the power of managed code through Silverlight:
void Button1_Click(object sender, EventArgs e)
{
    // Get a new GUID
    Guid g = Guid.NewGuid();

    // Display the GUID in the page user interface
    HtmlElement label1 = HtmlPage.Document.GetElementById("Label1");
    label1.SetProperty("innerHTML", g.ToString());
}
Needless to say, the managed code is a member of the Silverlight page codebehind class. The effect of the operation can be reflected both in the HTML of the host page (as in the preceding example) or directly in the Silverlight user interface—it is entirely up to you and your circumstances.
The attachment of the event is an operation that occurs through the browser interoperability layer and ends up calling the AttachEvent method on DOM objects. When the browser triggers the page-level event, a call is made back to Silverlight to execute the managed code.
When is such a feature helpful in the real world? Silverlight is a product designed to serve up a rich Web front end. You need Silverlight if you need managed code in the browser to do things that a managed language does better and faster than script. Examples of this are code-intensive operations where the speed of a compiled language beats script hands down, or operations that are not available in a functionally limited environment like the browser's. For sure, you don't strictly need Silverlight if all that you do is manipulate the DOM. So the ability to handle events in managed code is a very good feature to have, but it does require Silverlight. And you probably wouldn't want to engage Silverlight only for handling events.
In Silverlight 2, the HtmlWindow object provides the managed representation of the JavaScript window object. It stores a reference to the DOM object and allows you to drive it with a set of managed methods: Alert, Confirm, Prompt, Submit, Navigate, and even Eval. An instance of the HtmlWindow object is exposed to Silverlight developers through the Window property of the HtmlPage class. The following shows how to display a browser's message box from Silverlight:
HtmlPage.Window.Alert("Hello, world");
Let's explore what happens under the hood of this simple piece of code. The model I'll examine repeats itself in almost all methods on the HtmlWindow class:
public void Alert(string message)
{
    HtmlPage.VerifyThread();
    if (message == null)
    {
        message = string.Empty;
    }
    this.Invoke("alert",
                new object[] { message });
}
After the test on the UI thread, the method fixes the message string if it is null and proceeds to call the Invoke method on the base ScriptObject class. The Invoke method is the bridge between the managed world of Silverlight and the browser.
The method accepts two parameters, as you see here:
public virtual object Invoke(string name, params object[] args)
The first argument indicates the name of the method to invoke on the scriptable object stored in the current instance of ScriptObject. The second argument is simply the list of arguments for the method to invoke.
The method is responsible for correctly marshaling types across the two different runtime environments. On the way to the browser, it transforms managed objects into JavaScript-compatible types; on the way back, it does the reverse.
There's another method on HtmlWindow that deserves some attention here—the CreateInstance method:
public ScriptObject CreateInstance(
    string typeName,
    params object[] args)
The method allows you to create an instance of the specified JavaScript object. The type name parameter indicates the name of the JavaScript object to instantiate. Internally, the method prepares a dynamic JavaScript function that creates the specified object and then invokes the function from Silverlight:
ScriptObject xhr = HtmlPage.Window.CreateInstance("XMLHttpRequest");
The method CreateInstance is useful for obtaining a scriptable reference to a JavaScript object. Next, you script it using the Invoke method. These features are incredibly helpful when you really need to make synchronous calls from Silverlight 2. The problem is that synchronous calls are not permitted in Silverlight 2.

Synchronous Calls and Silverlight 2
Silverlight 2 supplies a variety of APIs to make calls to remote endpoints, but all of them must be asynchronous. You can start the call on a background thread, but you can't force the UI thread to sync up. If you block the UI thread on a sync object, you actually stop the UI thread indefinitely, and no command that resets the sync object can ever resume it. There's a big debate in the community about synchronous calls in Silverlight.
Nonetheless, synchronous calls from within the Silverlight API to a remote endpoint are simply not supported today because they are known and proven to have associated latency. Synchronous calls are an important feature in the browser environment, however. They're supported in all browsers that support XmlHttpRequest.
XmlHttpRequest is a browser object that enables AJAX scenarios. The object leverages the browser machinery to make a call to an endpoint within the same domain. By default, XmlHttpRequest operates asynchronously, and this is the way in which most AJAX frameworks use it. However, XmlHttpRequest can be quite easily configured to operate synchronously.
By creating and controlling an instance of XmlHttpRequest from Silverlight, you can arrange synchronous calls and fix all those particular scenarios where a synchronous call would make coding much easier. (Personally, I'm not a big fan of the async-only nature of Silverlight remote calls. Async calls are sufficient most of the time, but I think that, especially when you're adapting some existing front end to Silverlight, you may run into situations where a synchronous call would save you a lot of redesign. I'm the first to say that design is key, but if a consciously written synchronous call can save me hours or days of work, I'll definitely go for it.)
That said, the Silverlight team has good reasons to push the async-only approach because synchronous calls to remote endpoints could likely cause Silverlight applications to freeze the user interface, thus deteriorating the end user's experience with his browser and Web applications. Synchronous calls are technically possible, but the team does not support this natively in the platform in the interest of all applications and their consumers. Likewise, the team does not recommend resorting to a manually written synchronous remote call using XmlHttpRequest, as in Figure 5.
Figure 5 Making Synchronous Calls from within Silverlight 2
private void Button1_Click(object sender,
    System.Windows.RoutedEventArgs e)
{
    string url = "...";
    ScriptObject xhr = HtmlPage.Window.CreateInstance("XMLHttpRequest");
    xhr.Invoke("open", "POST", url, false);
    xhr.Invoke("setRequestHeader", "Content-Type",
        "application/x-www-form-urlencoded");

    // Prepare the body as the endpoint expects it to be
    string body = "...";
    xhr.Invoke("send", body);
    string response = (string) xhr.GetProperty("responseText");

    // Process the response and update the UI
    ProcessData(response)
}
Figure 5 shows how to use XmlHttpRequest to set up a synchronous same-domain call to a URL from within Silverlight 2. The key statement is when you invoke the open method on XmlHttpRequest. The Boolean argument indicates whether the call has to be asynchronous. If false, you instruct the object to proceed in a synchronous manner.
The drawback of this trick is that it leverages a low-level tool that just doesn't offer any facilities for converting strings to and from, say, JavaScript Object Notation (JSON) streams. If you employ this trick, you have to take care of any JSON serialization and deserialization using the DataContractJsonSerializer class.

XAML, Managed Code, and JavaScript
JavaScript functions can gain access to the content of the Silverlight application and perform read and write operations. The content of the Silverlight application is the tree of XAML elements.
Anything in the XAML document that is characterized by a unique name—the x:Name attribute—can be accessed and scripted in JavaScript. The first step entails getting a DOM reference to the Silverlight plug-in. In an ASP.NET AJAX page, you would use the following code:
var plugin = $get("SilverlightControl1");
SilverlightControl1 is the ID of the Silverlight control or the ID you used for the <object> tag that points to the downloadable content. Next, you point to the actual XAML content using the content property. To locate specific XAML elements, you use the findName method:
// Retrieve the XAML element tagged with the name of TextBlock1
var xamlTree = $get("SilverlightControl1").content;
var textBlock1 = xamlTree.findName("TextBlock1");

// Modify the current content of the text block element
textBlock1.Text = "...";
If you use JavaScript code to drive the content of the XAML document, then it is recommended that you cache in your page any reference to XAML elements that you encounter along the way. This would save you from repeatedly traversing the XAML tree to find the same element over and over again.
You should notice that using JavaScript to access the content of the XAML document is a bit outdated for Silverlight 2. It remains a good option if you are targeting Silverlight 1.0 or, at least, if you are serving up Silverlight applications consisting of only XAML and script code. If you can use managed code to decide about the content to display, then it is hard to imagine any changes that you might want to make to the user interface using JavaScript instead of managed code.
In Silverlight 2, you can use JavaScript code to invoke managed code. This option addresses a scenario where you have an HTML-based page powered by some managed code that performs critical operations.
Your JavaScript code can script any managed object hosted in Silverlight 2 that has been previously registered as scriptable:
guidHelper = new GuidHelper();
HtmlPage.RegisterScriptableObject("GuidTools", guidHelper);
The method RegisterScriptableObject takes the informal name of the object being registered and a variable instance. The string being the first argument is the name that the caller JavaScript code will use to refer to the registered object.
With respect to the preceding code, the following shows what you do from JavaScript in order to invoke a method on the sample class GuidHelper:
// Invoke a method on a scriptable managed object
var guid = $get("Silverlight1").content.GuidTools.Generate();

// Update the user interface with the results
$get("Random1").innerHTML = guid;
The public name of the scriptable object—in this case, GuidTools—is used as a member to access the underlying object from JavaScript. The pseudo property, GuidTools, is exposed by the content property of the Silverlight plug-in. Let's have a look at the GuidHelper class:
public class GuidHelper
{
    [ScriptableMember]
    public string Generate()
    {
        Guid g = Guid.NewGuid();
        return g.ToString();
    }
}
The ScriptableMember attribute on public methods of a scriptable class indicates that the method can be invoked from JavaScript. If all public methods of a class are scriptable, you can use the ScriptableType attribute on the class to automatically extend the scriptability attribute to all public methods.
An object registered as scriptable is instantiated through managed code and an existing instance is passed down to JavaScript. Not all managed types can be directly instantiated in JavaScript.
Before you can create an instance of a managed class from JavaScript, you first need to register the type you want as available for creation from JavaScript code:
HtmlPage.RegisterCreateableType("StockPicker", typeof(Samples.Order));
The first argument indicates the alias for the type to be used from within JavaScript. Here's how you create an instance of the Samples.StockPicker class:
var plugin = $get("Silverlight1");
var type = "Samples.StockPicker";
var inst = plugin.content.services.createObject(type);
You retrieve the Silverlight plug-in, access the content.services property and then invoke the method createObject. The argument to createObject is the string with the name of the managed type to instantiate.
A scriptable type that has complex and custom types in the signatures of its methods also works as a factory for its types. For example, imagine a managed type Customer with the following method:
void Add(Order order);
What about the Order type? Should it be declared createable? Not necessarily. If you omit the call to RegisterCreateableType, then you can only use as the factory a scriptable type that has a scriptable member that requires that type. In other words, if Customer is registered as scriptable and method Add is scriptable, then you can instantiate Order from JavaScript without declaring the type creatable. The following code would work:
var customer = $get("Silverlight1").content.Customer;
var order = customer.createManagedObject("Order");
order.ID = 1;
o.OrderDate = new Date();
customer.Add(order);
Managed types are marshaled down to JavaScript wrapped as browser objects. The wrapper browser object contains a table of valid methods to call from JavaScript. The list includes all methods declared as scriptable, plus createManagedObject. Invocations on methods are then resolved by invoking the corresponding method in the managed code.

Cross-Domain Access and Silverlight Plug-Ins
A Silverlight application that exposes a public managed API is subject to cross-domain access. If cross-domain access represents a security hazard, or just an unwanted feature, you should take your countermeasures.
When a page with a Silverlight plug-in is hosted in a frame, it is possible that the host page lives in a different domain than the Silverlight application. This means that the JavaScript code in the host page (cross-domain) is able to gain access through the frame to the Silverlight plug-in and script any public-managed object.
Cross-domain access to the Silverlight content may be disabled, fully enabled, or limited to scripting. It is disabled by default. You can control cross-domain access, as in Figure 6, by using the ExternalCallersFromCrossDomain attribute in the Deployment node of the Silverlight application manifest file. The manifest file is the file that is generated by Visual Studio 2008 when you compile a Silverlight project.
Figure 6 A Sample Manifest File for a Silverlight 2 App
<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment"
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
            EntryPointAssembly="BrowserFun"
            EntryPointType="BrowserFun.App"
            RuntimeVersion="2.0.30523.6"
            ExternalCallersFromCrossDomain="FullAccess">
    <Deployment.Parts>
        <AssemblyPart x:Name="BrowserFun"
                  Source="BrowserFun.dll" />
        <AssemblyPart x:Name="System.Windows.Controls.Extended"
                  Source="System.Windows.Controls.Extended.dll" />
    </Deployment.Parts>
</Deployment>
When you load a page in the browser that contains one or more Silverlight plug-ins, you still get one instance of the CLR per browser process. However, each running instance of the Silverlight plug-in gets its own AppDomain. Nothing is being shared between plug-ins and communication is possible, but only through code—and with the help of a little trick.
The trick consists essentially of using the browser interoperability layer as an intermediary. One plug-in exposes a managed interface and the other connects to it and invokes methods.
So one plug-in defines a few scriptable members, like so:
public partial class Page : UserControl
{
  public Page()
  {
    InitializeComponent();
    HtmlPage.RegisterScriptableObject(
        "Action", new ActionPageCommand());
  }
}
The ActionPageCommand class contains all methods that the plug-in exposes for external callers. Here's an example:
[ScriptableType]
public class ActionPageCommand
{
  public int GetRandomNumber()
  {
     Random rnd = new Random();
     return rnd.Next(0, 100);
  }
}
Scriptable members on a plug-in are directly visible to JavaScript code in the page. In Figure 7, you see a JavaScript wrapper for the publicly available interface of the Silverlight plug-in. Any plug-in interested in the interface of the other plug-in can create an instance of the JavaScript wrapper in the figure:
ScriptObject so = HtmlPage.Window.CreateInstance("ActionBar");
object result = so.Invoke("invokeGetRandomNumber");
Figure 7 Exposure to Another Silverlight Plug-In
<script type="text/javascript">
var ActionBar = function()
{}

function invokeGetRandomNumber$Impl()
{
    var plugin2Services = $get("Silverlight2").content;
    var results = plugin2Services.Action.GetRandomNumber();
    return results;
}

ActionBar.prototype =
{
    invokeGetRandomNumber: invokeGetRandomNumber$Impl
}
</script>
Is there a more direct way to invoke a Silverlight plug-in from another one? Yes, you can opt for the following:
HtmlElement plugin = HtmlPage.Document.GetElementById("Silverlight2");
var content = (ScriptObject) plugin.GetProperty("content");
var action = (ScriptObject) content.GetProperty("Action");
action.Invoke("GetRandomNumber");
The final effect is the same. This way, however, you're making more round-trips between Silverlight and the underlying browser.

In a Nutshell
The Silverlight Base Class Library includes facilities to connect to the browser and the host page and to read information and access the DOM. If you have a Silverlight-centric application, you probably don't need to do much with the host HTML page. However, if yours is a mixed solution with both ASP.NET AJAX and Silverlight, then the need to execute managed code from JavaScript or invoke JavaScript objects from within managed code is a real one.
The browser interoperability layer (also known as the HTML bridge) contains a lot of functionality to enable communication between the managed world of Silverlight and the interpreted world of JavaScript. Communication requires marshaling of types and objects across the layers. At the highest level of abstraction, you find a public API and documentation to explain the few things you need to know to invoke managed code from JavaScript and JavaScript objects from managed code.
If you have a special issue, or just want to split hairs as far as implementation details are concerned, you might want to keep an eye on Wilco Bauwer's informative blog.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and the coauthor of Microsoft .NET: Architecting Applications for the Enterprise (Microsoft Press, 2008). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Security Briefs
Threat Models Improve Your Security Process
Michael Howard
Contents
Threat Modeling
Code Review and Fuzz Testing Priority
Fuzz Testing
Attack Surface Reduction
Reducing Privileges
What Does It All Mean?
This column proposes a way to think about secure design from a more holistic perspective by using threat models to drive your security engineering process, primarily helping you prioritize code review, fuzz testing, and attack surface analysis tasks.
As a setup for this column, you might want to first read Jeremy Dallman's "'Crawling' Toward SDL" post, which can be found on the Security Development Lifecycle (SDL) blog. You will notice that the concepts in this column map very well onto Jeremy's ideas and will allow you to add more structure and precision to your security efforts as you learn to crawl, then walk, and finally run with the SDL.
What I am proposing is using the threat model to help drive other SDL security requirements, primarily code review priority, fuzz testing priority, and attack surface reduction. That's all there is to it. Of course, I need to explain myself, so let's look at each in a little detail. Let me start with threat modeling.

Threat Modeling
Many MSDN Magazine readers are familiar with the concept of threat modeling. Adam Shostack wrote an excellent article on the subject in the July 2008 issue ("Reinvigorate your Threat Modeling Process") and offers a glimpse of the future of the threat modeling process in more depth on the SDL blog.
In my opinion, when people think of security vulnerabilities, most think of implementation bugs. One could argue that the SDL is focused a little too much on implementation bugs, and historically it was because most of the vulnerabilities Microsoft has fixed resulted from implementation bugs. But over the last couple of years we have moved more resources to focus on secure design, in part because the implementation bugs are now more scarce thanks to the SDL.
Threat modeling is a cornerstone of the SDL because it allows a development team to think about secure design in a structured way. The threat modeling process can be effectively simplified into the following tasks:
  • Draw a picture of your software's data flows.
  • Use the "STRIDE per element" approach to find threats that apply to the design.
  • Address each threat.
  • Verify that you've modeled enough of the software, considered each threat, and addressed all the threats you discovered.
A core element of a threat model is the delineation of application entry points. The threat model captures entry points as trust boundaries during the "picture-drawing" phase. Examples include networking entry points, file and registry entry points, and so on.
A good threat model should also capture the network accessibility and authentication/authorization requirements of the interfaces. This includes network accessibility through IP address (local and remote, local subnet, and local-only access). It also includes authentication and authorization levels, anonymous access, user access, and administrator-only access. In the case of Windows access control lists (ACLs), authorization levels are finer-grained, but there is no need to go too deep (for a deeper discussion of ACLs, check out John Michener's article, "Understanding Windows File And Registry Permissions", in this issue. Keep it simple.
Another critical piece of data captured by the threat model is process identity. An entry point is simply an interface to a piece of code running in a process, and high-privilege processes are very dangerous if compromised. In Windows, the highest privilege processes are those running as SYSTEM or administrator. In Linux or Mac OS X, processes running as root are the most privileged.
Each app entry point should record the following data:
  • Entry point name
  • Network accessibility
  • Authentication and authorization
  • Process name and process identity
  • Comments (if needed)
Figure 1 shows a few examples. Clearly, the first network port is the most exposed because it is exposed to anyone on the planet. The most exposed entry points are the entry points at greatest risk of attack because something that is remotely accessible can be accessed by anyone with a computer on the same network—in this case, probably the Internet. Something that is accessible by anonymous users is accessible by anyone with no authentication or authorization. So an entry point that is accessible anonymously and remotely can be accessed by absolutely anyone, and that includes an awful lot of bad guys.
Figure 1 Process Entry Points
Entry Point | Network Accessibility | Authentication and Authorization | Process | Comment
TCP/1234 | Remote | Anonymous | App.exe (SYSTEM) | Main request/response
\\.\pipe\admin | Remote | Admin | App.exe (SYSTEM) | Administrative named pipe
ncacn_ip_tcp 609b954b-4fb6-11d1-9971-00c04fbbb345 | Remote | Authenticated Users | User.exe (Network Service) | User's account information RPC interface
Config.xml | Local | Admin | App.exe (SYSTEM) | Application configuration file
The administrative named pipe is accessible only to remote (and local) administrators, which is correctly enforced using an ACL. It's important that you verify the ACLs on all Windows named objects, including named pipes.
You will also notice that many entry points are located in code running as SYSTEM. The combination of an unauthenticated, remotely accessible entry point and a process running as SYSTEM is the largest possible exposure you can have in Windows. If there is a security bug in this code, anyone can make it fail from anywhere on the planet. And exploited code would run as SYSTEM, meaning the exploit has total control of the computer. I'll cover more of this topic later in the column.
The degree of exposure helps drive code review and fuzz testing priority. That's discussed next.

Code Review and Fuzz Testing Priority
Think about this for a moment; you have a lot of code to review and test. But you cannot spend every waking moment for the next 10 years reviewing code because, at some point, you have to ship a product to customers. What this means is that you must prioritize your security efforts; therefore, you must give the riskiest components your first and deepest efforts (I recommend that you read the "Who Is the Bad Guy" sidebar for some tips on finding the riskiest components). This means reviewing and fuzzing the riskiest code first.
When reviewing code by hand, it is important to review the most exposed code first. For example, the TCP/1234 socket example might have code like that shown in Figure 2. Because the call to recv reads from an unauthenticated and remote network port, buf should be treated as toxic waste until it is verified. In security circles, this data is often considered tainted and should not be relied upon for any accuracy or structure.
Figure 2 A Potential Problem Buffer
SOCKET  remoteSocket = accept(listenSocket,NULL, NULL);
if (remoteSocket == INVALID_SOCKET) {
  PRINTERROR("accept()");
  closesocket(listenSocket);
  return;
}

// Receive data from the client
char szBuf[256];
memset(buf, 0, sizeof(buf));
nRet = recv(remoteSocket,
      buf,  // Danger!
      sizeof(buf),
      0);

if (nRet == INVALID_SOCKET) {
  PRINTERROR("recv()");
  closesocket(listenSocket);
  closesocket(remoteSocket);
  return;
}

// process data in buf
To make things a little easier during code review, create a table that includes the source code location and API call for the entry point as well as the data (see Figure 3). Now follow the data from the entry point API to the code that verifies or sanitizes the incoming data. If you have no code that verifies the data integrity and your code performs potentially dangerous operations on the incoming data, then you might have a coding bug at best and a severe vulnerability at worst. Regardless of the code behavior, you should never assume the data is well formed.
Figure 3 Analyzing Entry Points
Entry Point | Source File | API and Data
TCP/1234 | Readfromsocket.cpp | recv(,buf,)
\\.\pipe\admin | AdminConsole.cpp | ReadFile(,bCommand)
ncacn_ip_tcp 609b954b-4fb6-11d1-9971-00c04fbbb345 | UserInfo_i.c and UserInfo.idl | Various
Config.xml | Config.cpp | ReadFile(,bFileContents)

Fuzz Testing
Look back at that last line: "never assume the data is well formed." The goal of fuzz testing is to determine whether you do make assumptions and, if so, whether someone can cause your application to fail by violating those assumptions. I am not going to explain fuzz testing here because it has been covered numerous times in MSDN Magazine (for instance, see the article "Fuzz Testing: Create a Custom Test Interface Provider for Team System" by Dan Griffin, but I will take a look at how you could be using fuzz testing.
Just like code review, you cannot fuzz test every possible code path, so you need to prioritize by exposure. Therefore, you should fuzz the most exposed entry points first. In my little example, I would fuzz test the TCP/1234 entry point brutally, early, and often. In fact, I would never stop fuzzing that entry point! CPU time is pretty cheap, after all.
I would go one step further and ask myself, why is this interface so highly exposed? Can I reduce the exposure by default? And that is the art of attack surface reduction.

Attack Surface Reduction
I have said this a million times before, and I'll say it again: attack surface reduction is as important as trying to get the code right because you'll never get the code right. I explained this mantra in more detail in my November 2004 article, "Attack Surface: Mitigate Security Risks by Minimizing the Code You Expose to Untrusted Users". I'll give you the short version here: you should reduce attack surface in order to reduce the potential risk to your customers. After all, an interface that is exposed to all attackers is a greater risk than an interface exposed only to attackers on your local subnet simply because there are fewer potential attackers in the second scenario.
Looking at my example, I would make the TCP/1234 entry point authenticated by default and allow only authenticated users access to it. This could be as simple as using Basic authentication (yes, I know it's a lousy authentication scheme) or something more complex, such as Kerberos. This would be my default, and if the system administrator wants to change it to anonymous access, then so be it. But that's a decision the administrator should make.
It's important to point out that if I do add authentication to the application, then the threat model needs to be updated because there is now an authentication process and an authentication credential database somewhere in the system.
Who Is the Bad Guy?
Here is the golden rule of security code review: always understand what the bad guy controls. A corollary to this is, Who is the bad guy? Who the bad guy is depends on the entry point's level of authentication and authorization.
In the case of the first TCP port example given in Figure 1, the bad guy is absolutely anyone on the planet. In the case of the second entry point, the named pipe, the bad guy isn't really a bad guy because he or she is a member of the local administrators group. If you can't trust your administrators, who can you trust?
Given that you have a finite amount of time, which interface should you code review earliest and deepest? Clearly, it's the first interface because anyone on the planet can access it. If you have a security bug in that code, anyone can compromise it. To really hammer the point home, think of a worm: worms rely on unauthenticated and remote networking interfaces.

Reducing Privileges
Another component of attack surface reduction is to reduce application privileges, and that's what I want to discuss next. Remember the processes that run as SYSTEM? Well, you need to think about using lower-privileged accounts to reduce the potential damage from a compromise. This is another important part of attack surface reduction.
Start with the most exposed entry points. In this case, if the decision is made to allow anonymous access to TCP/1234 by default, then you should ascertain why the process listening on that port needs to run as SYSTEM. Can it be reduced to a lower privileged account?
Also notice that the two network entry points and one file entry point all run in the same process. Is that really appropriate? Two entry points are for administrators, and one is for anonymous users. You should consider breaking this process up into two distinct processes so that you can take advantage of lower-privileged process identities.
If you look at the IIS 6.0 and IIS 7.0 architectures, you will notice that the processes handling anonymous and remote user requests (w3wp.exe) runs as Network Service, and the main administrative process (inetinfo.exe) runs as SYSTEM. The latter process is limited to administrators. In IIS 4.0 and IIS 5.0, the process handling and the administrative access were often performed by one high-privileged process.
Interestingly, a couple of aspects of attack surface reduction have been known for a long time. They are part of the classic Saltzer and Schroeder secure design principles and include least-privilege and least-common mechanism.
That second concept, least-common mechanism, needs a little explanation. It means you should reduce the amount of code shared by more than one user or user type. In my example, a single process performing administrative tasks and anonymous user requests can be dangerous. The remediation is to follow what IIS did and split the application into two processes that perform distinct tasks as separate identities.

What Does It All Mean?
To be absolutely honest, this column doesn't define anything new. It's all in the SDL! But the column explains the implicit relationship among the many security disciplines we define, describe and require in the SDL and how a good threat model can help streamline much of the rest of the security work required to ship more secure products to your customers.
Build a good threat model, determine your exposure, use a list of entry points (ranked by exposure) to help drive code review and fuzz testing priority, and finally drive your attack surface down. This will lead to more secure software.
Are you interested in finding out just how much you know about security? Take the challenging "Security IQ Test" found in this issue.

Send your questions and comments to briefs@microsoft.com.

Michael Howard is a Principal Security Program Manager at Microsoft focusing on secure process improvement and best practice. He is the coauthor of many security books including Writing Secure Code for Windows Vista, The Security Development Lifecycle, Writing Secure Code, and 19 Deadly Sins of Software Security.


Test Run
Group Determination In Software Testing
Dr. James McCaffrey
Code download available at: TestRun2008_11.exe (153 KB)
Browse the Code Online
Contents
Pure Plurality and Majority Runoff
Borda Count System
Condorcet Technique and Schulze Method
The Schulze Method in Code
More to Evaluate
Consider the general problem of determining a best option from a list of choices, where the decision-making process is performed by a group of people rather than by a single person or by a purely quantitative technique. Examples of this type of activity include the residents of a state voting for the state's governor from a list of candidates and the board of directors of a large company determining which of several smaller companies to target for acquisition. In essence, the group must rank its options in some way so that the best one can be determined.
Group determination of a best alternative occurs in software development and testing, too. For example, consider a group of beta users choosing the best user interface from a set of prototypes. Or imagine the members of an open source project voting for a policy.
I have observed that the practical techniques for performing group determination of a best alternative, and the potential pitfalls of the process, are not widely known in the software testing community. There are dozens of group analysis techniques. In this month's column I describe five of the most common methods used for group determination of a best alternative in a software testing environment: the pure plurality technique, the majority runoff technique, the Borda count system, the Condorcet method, and the Schulze method.
All of these analysis techniques are well-known outside of software engineering and have been extensively researched. I describe each technique with an emphasis on its application to software testing. I think you'll find knowledge of group analysis techniques a valuable addition to your toolkit.
Situations where exactly one of a set of alternatives must be selected by a group of people occur in a wide range of problem domains. Different problem domains tend to use different terminology. In sociology, for example, the study of this type of problem is usually called social choice theory and typically uses terms such as options and evaluators. In political science, analysis of these problems is often called voting theory and uses the terms candidates and voters. In mathematics, group determination of a best alternative is frequently considered a sub-branch of decision theory. I will use a mixture of terminologies in my discussion because there is no specific terminology set used when these techniques are applied to a software development and testing environment.
The techniques I describe in this column apply to a fairly narrow range of problems and do not apply strictly to quantitative scenarios where a best choice is clearly defined. As you'll see, when a group of people select a "best" alternative, the meaning of best is defined by the technique used. Also, the techniques I describe here do not apply to collaboration scenarios where a group of people use discussion and negotiation to arrive at a consensus decision.

Pure Plurality and Majority Runoff
Let's begin the discussion of group determination of a best alternative with a look at the simplest and most common technique—but let me warn you that this is not the best approach in many testing situations. Suppose you are developing an enterprise software application for your company's internal use and you create four quite different user interface prototypes. Although you could ask a single person to evaluate the different prototypes, in many cases a better approach is to ask a group of people to rank the four different designs.
It turns out that the process of performing a group evaluation and interpreting the results is somewhat trickier than you might expect. The simplest possible group evaluation technique is to submit the choices to the evaluators and allow them to choose, or vote for, just one alternative. The alternative that receives the most votes is declared the winner. This technique is often called the pure plurality rule.
So, suppose you submit your four different prototypes (let's call them A, B, C, and D) to 10 evaluators and ask each person to choose the best prototype. And suppose the results are: Prototype A is best according to 4 people, prototype B is ranked best by 3 people, prototype C is chosen by 2 people, and prototype D is selected best by 1 person. Therefore, you select prototype A for further development.
There are several problems with this. Suppose that while your evaluators are looking at your four prototypes, they either implicitly or explicitly rank each choice, as shown in Figure 1. In this diagram, each column represents one evaluator's ranking of the prototypes.
Figure 1 Prototype Rankings for 10 Evaluators
Notice that although prototype A is selected by the pure plurality rule, that prototype is evaluated as the worst by 60% of the evaluation group. Additionally, in this situation, suppose prototype A was compared in turn against each of the other prototypes. Prototype A would lose each head-to-head comparison by a 60% to 40% margin. A second, closely related problem with the pure plurality technique is that in some sense the non-winning opinions do not contribute directly to the final result.
The problems with pure plurality lead to another group selection technique usually called the majority runoff system. There are several variations on the technique, but the winner is required to gain more than 50% of the first-place votes of all votes cast.
This can be accomplished by having multiple rounds of voting or performing just a single round of voting but requiring the evaluators to rank all options. For example, if evaluators vote for the best prototype design according to the data in Figure 1, no prototype would have a majority after the first round of voting. So the list of candidates could be reduced, usually to the top two options; in this case, prototype A with four votes and prototype B with three votes.
A second round of voting then takes place. If at that time evaluators vote according to the ranking preferences in Figure 1, after the second round of voting prototype A would still receive four votes as best, but prototype B would receive six votes because the evaluators who originally selected prototypes C and D would now select prototype B.
In my experience, using multiple rounds of voting in a software development and testing environment has significant disadvantages compared to using a single round. For one thing, in some software development scenarios, multiple rounds of voting simply aren't feasible. Additionally, because it entails dealing with human beings, multiple rounds of voting can get tedious. Worse, situations can arise where a group of people are voting for losing alternatives over and over (receiving negative feedback each time) until they are forced to choose an option they don't really like. If the voters are stakeholders in the sense that they will be affected by the outcome of the voting, you may end up with disenfranchised voters at the end of a physical runoff scheme.
Instead of performing multiple actual rounds of voting where each evaluator votes for just one alternative, you can simply perform one round of voting but require evaluators to rank all candidates from best to worst. This allows you to determine voters' preferences in virtual rounds of voting.
Requiring ranking of all choices has certain disadvantages, too. The technique often isn't practical when the number of options is large (typically more than eight), simply because humans have difficulty comparing large numbers of alternatives. Also, research has shown that evaluators tend to get careless with their evaluations of options that are low on their list of preferences. And, when performing multiple physical rounds of voting, evaluators may not rank a subset of alternatives in the same order in which they rank the entire set of alternatives.
Schulze Method: Pros and Cons
The Schulze method has strengths and weaknesses. One weakness in some scenarios is that Schulze is far from intuitive. In situations where evaluators are stakeholders, the Schulze method can easily be viewed as a mysterious black box technique that produces a magic result.
The primary advantage of the Schulze method is that it has been extensively analyzed and has been shown to meet many favorable criteria. (Recall that the Borda count system can be affected by the removal of an irrelevant alternative.)
There are many voting-system criteria. For example, the monotonicity criterion can be loosely stated as the principle that a winning option cannot become a non-winner by one or more evaluators ranking that option higher. The independence of clones alternatives criterion is, stated loosely, that adding a new option that is identical to an existing option should not change the outcome of an analysis. There are many proposed voting-system criteria, and the Schulze method has been shown to meet most of these criteria as well.
An interesting exception is that the Schulze method violates what is called the participation criterion. In informal terms, it is possible to add to an existing system, new voters who prefer the current winner to some other alternative, which results in the less-preferred alternative becoming a Schulze winner.
In spite of its shortcomings, the Schulze method is often very effective, especially in situations with a large, educated group of evaluators (meaning they have knowledge of the underlying Schulze method) who are determining a policy alternative (as opposed to a product decision) and who are stakeholders in the final result. For example, the Schulze method is used by several open source groups to determine general policy decisions and to elect officers. In software testing scenarios, the Schulze method is generally the first technique I will choose to employ, and then I will use those results as a baseline against the results of other techniques.

Borda Count System
The problems with the pure plurality and majority runoff techniques led to the development of two group analysis techniques that are very common. These techniques, the Borda count system and the Condorcet pair-wise comparison method, are named after the two mathematicians who proposed the approaches in the late 1700s.
The Borda count approach is very simple and one with which you are almost certainly familiar. Suppose there are n alternatives. Each evaluator ranks the alternatives from best to worst. A top-ranked alternative is assigned a value of n-1 points, a second-ranked alternative is assigned n-2 points, and so on, down to the last-ranked alternative, which receives 0 points. The Borda count is just the sum of the point values for each alternative.
A quick example will make the ideas clear. Suppose there are four choices, A, B, C, and D. In this case, a first-place ranking is worth 3 points, second place is worth 2, third place is worth 1, and last place is worth 0 points. Figure 2 shows the results and then, underneath, the Borda counts for seven evaluators where alternative A is selected as the best option.
Figure 2 Results and Borda Counts for Seven Evaluators
The Borda count technique is widely used in many problem domains, especially in sports competitions. Advantages of the Borda count technique when used in a software development and testing environment are that the technique seems to make sense intuitively, and all evaluators' opinions are taken into account.
The Borda count system has several technical problems, however. Suppose that we decide to remove option D, which was a clear loser. Common sense suggests that removing an irrelevant alternative should not affect the final result. But Figure 3 shows what happens after removing option D. Now option B becomes the best alternative instead of option A. In other words, with this set of data, removing an irrelevant alternative changes the outcome. It is even possible to construct Borda count examples where the best alternative actually becomes the worst alternative. This effect is sometimes called the Borda count winner-becomes-loser paradox.
Figure 3 Results and Borda Counts after Removing Option D
There are two other closely related technical problems with the Borda count technique. The first is the pair-wise comparison problem. It is possible when using the Borda count technique to run into situations where the Borda winner would lose against every alternative in a head-to-head comparison.
Suppose, for example, that there are four alternatives and 12 evaluators (see Figure 4). Using the Borda count method, alternative A is selected as best with 25 points. However, if we look at head-to-head comparisons we notice that alternative B is preferred to all other alternatives: B is preferred to A by a score of 7 people to 5, B is preferred to C also by 7 to 5, and B is preferred to D again by 7 to 5.
Figure 4 Option B Is the Head-to-Head Winner
Another problem with the Borda count technique is that it is vulnerable to evaluator manipulation. It is theoretically possible to affect the outcome of a Borda count evaluation by adding a spurious alternative that is very close (in terms of evaluators' preferences) to an existing alternative. This can have the effect of diluting close alternatives and changing the winner of the analysis. The references at the end of this column contain examples of the effect.
Despite the Borda count technique's technical flaws, the system is quite practical and often works well for policy determination by small groups in a software testing environment. Based on my experience, the Borda count technique is generally seen by voters as fair in a subjective way, and, therefore, if voters are stakeholders, the people who voted for non-winning alternatives generally accept the result of the voting and do not come away with a psychological bias against the winning alternative.
The moral of the story here is that if you do use the Borda count technique, be alert for, and check your data for, the effect of removing an irrelevant alternative, the result of pair-wise comparisons between the Borda count winner and non-winning alternatives, and the mid-process addition of a new alternative.

Condorcet Technique and Schulze Method
The Condorcet technique for collaboratively determining the best option from a set of choices was developed primarily as a reaction to the head-to-head paring problem of the Borda count method. The Condorcet method is very simple. It requires evaluators to rank all alternatives, and then a comparison of the results between each possible pair of alternatives is performed.
If one alternative beats every other alternative in a head-to-head comparison, then that one alternative is declared the Condorcet winner. If there is no Condorcet winner, then a separate, tie-breaking technique is employed. The tie-breaking technique can be any other technique; the point of the Condorcet system is that a unanimous head-to-head winner should automatically win over any alternatives chosen by any other criteria.
In practice, the Condorcet system is not often used by itself as a group evaluation technique. Instead, evaluation techniques such as pure plurality and Borda count are mathematically analyzed to see if they satisfy the Condorcet principle in all cases. So, instead of reading something like, "We decided to use the Condorcet system to evaluate our alternatives," it is more usual to read something like, "We decided to use the XYZ evaluation system, which satisfies the Condorcet principle."
One of the most interesting systems that satisfies the Condorcet principle has a variety of names, including the Schulze method, the clone-proof Schwartz sequential dropping technique, and the beat-path method. I will call this technique the Schulze method.
The Schulze method is somewhat more complicated than the other techniques I've described so far. It is similar to the Condorcet system in the sense that Schulze looks to see if one option dominates all other options. However, instead of directly using raw ranking data, as the Condorcet system does, the Schulze method performs a head-to-head comparison using an indirect measure of the strength between alternatives. This indirect measure is called the path strength.
Let's walk through an example. Take a look at Figure 5. It shows a simple demo of the Schulze method in action. The first part shows the example input. There are four options (A, B, C, and D) and 11 evaluators. The first line means that 3 evaluators rank option A better than B, which in turn is better than C, which is better than D. The Schulze method allows evaluators to rank options equal to each other and also allows evaluators to leave out alternatives altogether.
Figure 5 Schulze Method Demonstration
The next part of the output shows a matrix of pair-wise defeats. This defeats matrix is computed directly from the original ranking data.
The first row of the matrix means option A is preferred by 6 voters over option B, preferred by 3 voters over option C, and by 4 voters over option D. The second row of the matrix means option B is preferred by 5 voters over option A, by 7 voters over option C, and by 8 voters over option D. The third and fourth rows, for options C and D, respectively, are interpreted similarly. The pair-wise defeats matrix has 0 values on the main diagonal because these values are comparisons between an option and itself.
At this point the Condorcet system would check this direct pair-wise data to see if there is a unanimous head-to-head winner, but the Schulze method derives an indirect measure of strength between alternatives. The next part of output is the strength of paths between each option. This is the key idea behind the Schulze method and is a very clever notion. The third row of this matrix means the strength of the path from option C to option A is 8, the strength from C to B is 4, and the strength from C to D is 10.
Explaining the idea of path strength is best done visually, as shown in Figure 6. First we express the raw ranking data as a matrix of pair-wise defeats. Next we can conceptualize the pair-wise defeats data as a directed graph.
Figure 6 Pair-Wise Defeat Data in Matrix and Graph Forms
The arrow from node A to node B means that option A is preferred by 6 evaluators over option B. I do not put an arrow from node B to node A because option B is preferred by only 5 evaluators over option A, so A dominates B. In this example, all the arrows are unidirectional, but if two nodes are preferred by equal numbers of evaluators, then I can draw a bidirectional arrow.
Now consider the path between node A and node D. A beats B by 6, and B in turn beats D by 8. We chose 6, the smaller of these two values, to represent the overall strength of path between A and B. The idea is that the overall strength between two nodes that have intermediate nodes is best represented by the smallest direct node-to-node strength. Notice that we can also take a different path from A to D and say that A beats B by 6, and B beats C by 7, and C beats D by 10, and select the smallest number, 6, which would give the same result.
It turns out that computing the strength of each pair of nodes can be performed by a variation of the elegant Floyd–Warshall algorithm, as I'll explain in a moment. The standard Floyd–Warshall algorithm finds the shortest paths in a weighted, directed graph. Once the path strengths have been determined, we can determine the Schulze method winners by checking to see if there is any option where its path strength is greater than or equal to all other options' corresponding path strengths.
Figure 7 shows the matrix of path strengths that corresponds to the data in Figure 6 and demonstrates that option B is a Schulze method winner. Visually, an option is a Schulze winner if every path strength value on that option's row is greater than or equal to the corresponding value across the matrix's main diagonal.
Figure 7 Path Strength Data Used to Determine a Schulze Winner
Option B is the winner because option B's path strength to option A is 7 (but option A's path strength to B is just 6), option B's path strength to option C is 7 (but C's path strength to B is just 6), and option B's path strength to option D is 8 (but D's path strength to B is just 6). On the other hand, option D is not a Schulze winner because option D's path strength is better than option A (7 to 6), but option D's strength is worse than option C (6 to 10).

The Schulze Method in Code
In this example there is just one Schulze winner, but it is possible for there to be multiple Schulze winners. In situations like these you would have to use some form of tiebreaker. Let's go over some of the code in the program that generates the output in Figure 5 to help explain the Schulze method.
The program begins by setting up an enumeration type to represent the alternatives and a global value N to hold the actual number of alternatives:
enum options { A, B, C, D, E, F, G, H, I, J };
   static int N; // actual number of options
Then the demo program sets up problem input as a string array:
string[] rawData = new string[] {
           "A>B>C>D 3",
           "B>C>D>A 4",
           "C>D>A>B 2",
           "D>C>B>A 1",
           "C>A>B>D 1"
                          };
For simplicity, my demo program assumes that ties are not allowed and that all options are ranked. Doing so lets me parse the raw data strings very easily. The complete demo program is available in the code download that accompanies this column. You should be able to modify the code to allow ties and missing options, without too much trouble.
Next, my demo program scans through the rawData string array and determines and prints the number of options and the number of evaluators. Then the programs rescans the input array and creates a matrix of pair-wise defeats data using a static helper method named MakeDefeats:
int[,] defeats = MakeDefeats(rawData);
Console.WriteLine("\n== Pairwise defeats data is: \n");
Print(defeats);
The defeats matrix is then passed as input to another helper method, named MakePathStrengths, which creates a matrix of path strengths:
int[,] strengths = MakePathStrengths(defeats);
Console.WriteLine("\n== Path strength data is: \n");
Print(strengths);
Inside the MakePathStrengths method, I begin by eliminating entries that are dominated:
int[,] result = new int[N, N];
for (int i = 0; i < N; ++i)
{
  for (int j = 0; j < N; ++j)
  {
    if (d[i, j] > d[j, i])
      result[i, j] = d[i, j];
    else
      result[i, j] = 0;
  }
}
Notice that the statement result[i, j] = 0; is not technically necessary because C# automatically initializes int arrays to all 0 values.
Next I use a variation of the Floyd–Warshall algorithm to compute the strength of each possible path:
for (int k = 0; k < N; ++k) {
  for (int i = 0; i < N; ++i) {
    if (k == i) continue;
    for (int j = 0; j < N; ++j) {
      if (k == j || i == j) continue;
      result[i, j] =
        Math.Max(result[i, j], Math.Min(result[i, k], result[k, j]));
    } //j
  } //i
} //k
If you go ahead and hand-trace through this code, you will see how it implements the determination of the path strength that exists between node [i] and node [j], as I described previously. The Max method in the code works to ensure that on each iteration through the outer loop I update the result matrix only if a stronger path happens to be found. The Min method in the code corresponds to the idea of using the smallest direct node-to-node strength to represent the overall path strength that is in existence between any two nodes.
After I have all the path strengths, I can use that information to compute an array that represents Schulze winners and then iterate through that array and display the enumeration value that corresponds to each winner:
bool[] winners = MakeWinners(strengths);
Console.WriteLine("\n== Best option(s) is: \n");
for (int i = 0; i < winners.Length; ++i) {
  if (winners[i] == true)
    Console.Write(Enum.GetName(typeof(options), i) + " ");
}
The MakeWinners method sets up a one-dimensional Boolean array where a value of true indicates a winner and a value of false indicates a non-winner. I assume every option is a winner, then iterate through the path strength matrix, comparing the strength value at index [i,j] with that at index[j,i] to see if that option is in fact a non-winner:
static int[] MakeWinners(int[,] ps) {
  bool[] result = new bool[N];

  for (int i = 0; i < N; ++i)
    result[i] = true;

  for (int i = 0; i < N; ++i)   {
    for (int j = 0; j < N; ++j)
      if (ps[i, j] < ps[j, i])
        result[i] = false;
  }
  return result;
}

More to Evaluate
Some of the key factors you should consider when using group evaluation techniques include the number of alternatives, the number of evaluators, whether the alternatives are policy decisions or product decisions, and the extent to which evaluators are affected by the final result of the analysis. In general, you should use more than one group analysis technique when possible in order to use multiple results to cross-validate your analysis (if multiple techniques yield the same conclusion, you gain some measure of confidence in your results).
The situations where the techniques I've described in this column are used are almost always quite subjective. Your experience and intuition must play a big part in analyzing group determination results.
As a general rule of thumb, the pure plurality technique is only effective in situations where evaluators are not stakeholders or when ranking all alternatives is not feasible. The majority runoff technique has the potential to alienate up to 49% of your evaluators, and based on my experience you should generally use this technique primarily to supplement other techniques.
The Borda count system, in spite of technical flaws, works quite well, especially when the evaluators are stakeholders in the analysis. The Condorcet principle is very simple and so can often be used to validate the results of other techniques. The Schulze method is an excellent general-purpose technique in many software testing situations, except when you are dealing with a relatively small group of evaluators who are influenced by the results and who do not understand the Schulze algorithm.
This column just scratches the surface of an interesting and useful subject. There is a large body of research on group techniques to determine the best alternative from a set of options. One good reference is the book A Primer in Social Choice Theory, by Wulf Gaertner (Oxford University Press, 2006). I also recommend Comparing Voting Systems, by Hannu Nurmi (Springer-Verlag, 1987).
You might find it interesting to compare the problem of group analysis to determine a best alternative to the related problem of rating a set of alternatives based on multiple attributes. If that's the case, I address two different multi-attribute decision techniques in the June 2005 (msdn.microsoft.com/magazine/cc163785) and October 2006 (msdn.microsoft.com/magazine/cc300812) installments of the Test Run column.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working for Microsoft in Redmond. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006), and he can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Wicked Code
Silverlight 2 Transforms And Clipping Regions
Jeff Prosise

This column is based on a prerelease version of Silverlight 2. All information is subject to change.

Code download available at: WickedCode2008_11.exe (51,861 KB)
Browse the Code Online
Contents
Under the Magnifying Glass
Round and Round She Goes (in 3D)
Live Video Overlays the Easy Way
Silverlight is the revolutionary platform from Microsoft for creating highly functional, immersive, interactive browser-based applications. Silverlight 2, which was in beta at the time of this writing but will be nearing release by the time you read this, supports multithreading, networking, browser integration, isolated storage, strong typing, reflection, and much more. But Silverlight is best known for stunning graphics.
Silverlight 2 combines a vector-based XAML rendering engine with browser-based versions of the CLR and the Microsoft .NET Framework Base Class Library. Armed with a working knowledge of XAML, developers and designers can make amazing things happen. Key to many of the eye-popping visual effects featured in Silverlight applications are transforms and clipping regions. The page-turn framework presented in the May 2008 installment of Wicked Code, ("Silverlight Page Turning Made Simple"), for example, relies heavily on transforms and clipping regions to create the illusion of pages turning in a browser the same way they turn in a book or magazine.
There are more cool tricks you can pull off with a little help from your friends Mr. Transform and Ms. Clipping, which you'll see in a minute. But first, be aware that the samples I'll present were built and tested with Silverlight 2 Beta 2 and thus may require modification to work with Silverlight 2 RC and RTM releases.

Under the Magnifying Glass
Windows Presentation Foundation (WPF) programmers sometimes use VisualBrushes to create virtual magnifying glasses like the one pictured in Figure 1. Silverlight doesn't support VisualBrush, which has led some developers to believe—even to pronounce—that it's impossible to create similar effects in a Silverlight application.
Figure 1 Silverlight Magnifier in Action
Fortunately, you can simulate magnifying glasses in Silverlight; doing so just requires a bit of ingenuity with transforms and clipping regions. The Magnifier application presented here demonstrates how. Figure 2 shows an abridged version of Magnifier's Page.xaml file. It declares two almost identical Canvases, each containing a copy of the content pictured in Figure 1. The first Canvas—MainCanvas—is the one that the user normally sees. The second Canvas—ZoomCanvas—contains the same content as the first but also includes a ScaleTransform that magnifies everything by a factor of 4.
Figure 2 Page.xaml
<UserControl x:Class="Magnifier.Page"
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <Grid x:Name="LayoutRoot" Background="Black"
    MouseLeftButtonDown="OnMouseLeftButtonDown" MouseMove="OnMouseMove"
    MouseLeftButtonUp="OnMouseLeftButtonUp">
    <Canvas x:Name="RootCanvas" Width="800" Height="800">

      <!-- Main canvas -->
      <Canvas x:Name="MainCanvas" Canvas.Left="0" Canvas.Top="0"
        Width="800" Height="900" Background="Black">
        <Canvas Canvas.Left="90" Canvas.Top="30" Width="620" Height="470">
          <Rectangle Canvas.Left="0" Canvas.Top="0" Width="620"
            Height="470" Fill="White" />
          <Image Canvas.Left="10" Canvas.Top="10" Width="600" Height="450"
            Source="Images/BobCat.jpg" />
        </Canvas>
        <Canvas Canvas.Left="90" Canvas.Top="540">
          <Line Canvas.Left="0" Canvas.Top="0" X1="0" Y1="0" X2="620"
            Y2="0" Stroke="#808080" StrokeThickness="3"
            StrokeDashArray="1,1" />
          <TextBlock Canvas.Left="0" Canvas.Top="10" Foreground="White"
            FontSize="36" Text="BVM BobCat" />
          <TextBlock Canvas.Left="0" Canvas.Top="70" Foreground="White"
            FontSize="12" TextWrapping="Wrap" Width="620" Text="..." />
          <Line Canvas.Left="0" Canvas.Top="180" X1="0" Y1="0" X2="620"
            Y2="0" Stroke="#808080" StrokeThickness="3"
            StrokeDashArray="1,1" />
        </Canvas>
      </Canvas>

      <!-- Zoom canvas -->
      <Canvas x:Name="ZoomCanvas" Canvas.Left="0" Canvas.Top="0"
        Width="800" Height="900" Background="Black" Visibility="Collapsed">
        <Canvas.RenderTransform>
          <ScaleTransform CenterX="0" CenterY="0" ScaleX="4" ScaleY="4"/>
        </Canvas.RenderTransform>
        <Canvas.Clip>
          <EllipseGeometry x:Name="Lens" Center="0,0"
            RadiusX="40" RadiusY="40" />
        </Canvas.Clip>
          ...
        <Path Canvas.Left="0" Canvas.Top="0" Stroke="#808080"
          StrokeThickness="1">
          <Path.Data>
            <EllipseGeometry x:Name="LensBorder" Center="0,0"
              RadiusX="40" RadiusY="40" />
          </Path.Data>
        </Path>
      </Canvas>
    </Canvas>
  </Grid>
</UserControl>
ZoomCanvas is normally hidden from the user, but when the left mouse button goes down, OnMouseLeftButtonDown (see Figure 3) makes ZoomCanvas visible by toggling its Visibility property. ZoomCanvas is never seen in its entirety. However, its Clip property is initialized with an EllipseGeometry that exposes content in a perfect circle that appears, for all intents and purposes, to be a magnifying glass. You can change the size of the magnifying glass—actually the clipping region—by modifying the EllipseGeometry's RadiusX and RadiusY properties. If you do, you'll also want to modify the same properties of the EllipseGeometry named LensBorder to match (that EllipseGeometry draws the border around the magnifying glass).
Figure 3 Page.xaml.cs
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Input;

namespace Magnifier
{
    public partial class Page : UserControl
    {
        private bool _dragging = false;
        private const double _scale = 4.0;

        public Page()
        {
            InitializeComponent();
        }

        private void OnMouseLeftButtonDown(object sender,
            MouseButtonEventArgs e)
        {
            double x = e.GetPosition(RootCanvas).X;
            double y = e.GetPosition(RootCanvas).Y;
            PositionLens(x, y);

            ZoomCanvas.Visibility = Visibility.Visible;
            ((FrameworkElement)sender).CaptureMouse();
            _dragging = true;
        }

        private void OnMouseMove(object sender, MouseEventArgs e)
        {
            if (_dragging)
            {
                double x = e.GetPosition(MainCanvas).X;
                double y = e.GetPosition(MainCanvas).Y;
                PositionLens(x, y);
            }
        }

        private void OnMouseLeftButtonUp(object sender,
            MouseButtonEventArgs e)
        {
            if (_dragging)
            {
                ZoomCanvas.Visibility = Visibility.Collapsed;
                ((FrameworkElement)sender).ReleaseMouseCapture();
                _dragging = false;            }
        }

        private void PositionLens(double x, double y)
        {
            Lens.Center = LensBorder.Center = new Point(x, y);
            ZoomCanvas.SetValue(Canvas.LeftProperty, (1 - _scale) * x);
            ZoomCanvas.SetValue(Canvas.TopProperty, (1 - _scale) * y);
        }
    }
}
You can see the magnifying glass in action by running the application and clicking and holding the left mouse button anywhere in the scene (even over the text at the bottom of the page). Also observe that moving the mouse with the button held down moves the magnifying glass, too. This is accomplished by modifying the Center properties of the two EllipseGeometries to reposition the lens and by adjusting the Canvas.Left and Canvas.Top properties of ZoomCanvas so that the content exposed through the clipping region coincides with the content under the cursor in MainCanvas. See the OnMouseMove and PositionLens methods to see exactly how this is done.
As you play with the application, note that the magnifying glass doesn't blow up pixels when it's positioned over the image; it reveals greater image detail. That's because the image's native resolution is 2,400x1,800 pixels. The image is declared in MainCanvas and ZoomCanvas with a width and height of 600 and 450, respectively, or exactly one-fourth its native resolution. But when ZoomCanvas expands the image four times using a ScaleTransform, the image reverts to its native size. Therefore, the image the user normally sees is compressed to one-fourth its actual width and height, but the image seen through the magnifying glass is displayed at its native resolution. If the image's native size were 600x450, the magnified view would be pixelated.

Round and Round She Goes (in 3D)
Another difference between Silverlight and WPF is the latter's support for 3D graphics. But that hasn't stopped Silverlight developers from employing 3D effects in their apps. The lack of 3D support simply means that Silverlight programmers have to work a little harder to create 3D effects. One such effect is demonstrated in the next example.
When it first appears, the SpinAndZoom application displays a photo of my youngest daughter leaping off a lava rock into a crystal clear ocean. But drag the mouse over the picture with the left button held down and the photo spins around a vertical axis to reveal a picture of my other daughter on the sea floor at the base of the same lava formation (Figure 4). Keeping your daughters underwater, by the way, is a great way to hide them away from boys.
Figure 4 Spinning a Photo around a Vertical Axis—in 3D
Figure 5 shows the XAML underlying the 3D spins. It declares four separate images: two for the front photo and its reflection, and two more for the back photo and its reflection. The reflections are generated by using a ScaleTransform to flip the image upside down and by combining an OpacityMask with a LinearGradientBrush to fade the reflected image as a function of distance. Spinning is accomplished by programmatically manipulating a ScaleTransform (SpinScaleTransform) and a SkewTransform (SpinSkewTransform). As the angle of rotation increases, the SkewTransform increases the angle of the top and bottom edges, while the ScaleTransform squeezes the image horizontally. Choreographed properly, the result is a convincing imitation of a 3D rotation.
Figure 5 Page.xaml
<UserControl x:Class="SpinAndZoom.Page"
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <Grid x:Name="LayoutRoot" Background="Black"
    MouseLeftButtonDown="OnMouseLeftButtonDown" MouseMove="OnMouseMove"
    MouseLeftButtonUp="OnMouseLeftButtonUp">
    <Canvas x:Name="SpinCanvas" Width="400" Height="300">
      <Canvas.RenderTransform>
        <TransformGroup>
          <ScaleTransform x:Name="SpinScaleTransform" CenterX="200" />
          <SkewTransform x:Name="SpinSkewTransform" CenterX="200" />
          <ScaleTransform x:Name="ZoomScaleTransform" CenterX="200"
            CenterY="150" />
        </TransformGroup>
      </Canvas.RenderTransform>

      <!-- Front -->
      <Image x:Name="Front" Source="Images/Abby.jpg" Canvas.Left="0"
        Canvas.Top="0" Width="400" Height="300" Visibility="Visible"
        Stretch="Fill" />

      <!-- Front reflection -->
      <Image x:Name="FrontReflection" Source="Images/Abby.jpg"
        Canvas.Left="0" Canvas.Top="500" Width="400" Height="200"
        Visibility="Visible" Stretch="Fill">
        <Image.RenderTransform>
          <ScaleTransform ScaleY="-1" />
        </Image.RenderTransform>
        <Image.OpacityMask>
          <LinearGradientBrush StartPoint="0,0" EndPoint="0,1">
            <GradientStop Offset="0.5" Color="#00000000" />
            <GradientStop Offset="1" Color="#80000000" />
          </LinearGradientBrush>
        </Image.OpacityMask>
      </Image>

      <!-- Back -->
      <Image x:Name="Back" Source="Images/Amy.jpg" Canvas.Left="0"
        Canvas.Top="0" Width="400" Height="300" Visibility="Collapsed"
        Stretch="Fill" />

      <!-- Back reflection -->
      <Image x:Name="BackReflection" Source="Images/Amy.jpg" Canvas.Left="0"
        Canvas.Top="500" Width="400" Height="200" Visibility="Collapsed"
        Stretch="Fill">
        <Image.RenderTransform>
          <ScaleTransform ScaleY="-1" />
        </Image.RenderTransform>
        <Image.OpacityMask>
          <LinearGradientBrush StartPoint="0,0" EndPoint="0,1">
            <GradientStop Offset="0.5" Color="#00000000" />
            <GradientStop Offset="1" Color="#80000000" />
          </LinearGradientBrush>
        </Image.OpacityMask>
      </Image>
    </Canvas>
  </Grid>
</UserControl>
The code responsible appears in the download and is excerpted in Figure 6. RotateTo, the key method, is called by the MouseMove handler to increase or decrease the angle of rotation as the mouse cursor traverses the scene. RotateTo employs a bit of trig to compute SkewTransform and ScaleTransform parameters based on the geometry in Figure 7. For a given angle of rotation, RotateTo computes the width and inner angle (the angle nearest the center) of the triangle highlighted in purple. The width tells RotateTo what value to set the ScaleTransform's ScaleX property to, and the angle tells it what to set the SkewTransform's AngleY property to.
Figure 6 RotateTo from Page.xaml.cs
private void RotateTo(double angle)
{
    double radians = (angle * Math.PI) / 180;
    double scaleX = Math.Abs(Math.Cos(radians));

    // Update the ScaleTransform
    // Avoid ScaleX == 0 to prevent images from disappearing!
    SpinScaleTransform.ScaleX = Math.Max(0.005, scaleX);

    // Update the SkewTransform
    if (angle != 90 && angle != 270) // Tangent undefined
    {
        double h = Math.Sin(radians) * _aspect;
        double r = Math.Atan(h / scaleX);
        if (angle > 90 && angle < 270)
            r = -r;
        SpinSkewTransform.AngleY = (r * 180) / Math.PI;
    }
}
Figure 7 The Geometry of 3D Spins
The key to the effectiveness of the illusion is that the vertices of the rotating image follow the track defined by an imaginary cylinder. The degree to which that cylinder is tilted toward the user is controlled by the private field named _aspect. You can increase the severity of the cylinder by increasing the value of _aspect.
SpinAndZoom also demonstrates how to implement another common visual effect found in Silverlight applications: interactive zooms. An event handler named OnMouseWheelTurned shown in Figure 8 responds to mouse-wheel movements by manipulating the ScaleTransform named ZoomScaleTransform. That transform scales every XAML object in the scene up or down. (Try it; position the mouse cursor anywhere in the scene and roll the mouse wheel back and forth.)
Figure 8 OnMouseWheelTurned from Page.xaml.cs
private void OnMouseWheelTurned(Object sender, HtmlEventArgs args)
{
    double delta = 0;
    ScriptObject e = args.EventObject;

    if (e.GetProperty("wheelDelta") != null) // IE and Opera
    {
        delta = ((double)e.GetProperty("wheelDelta"));
        if (HtmlPage.Window.GetProperty("opera") != null)
            delta = -delta;
    }
    else if (e.GetProperty("detail") != null) // Mozilla and Safari
    {
        delta = -((double)e.GetProperty("detail"));
    }

    if (delta > 0)
    {
        if (ZoomScaleTransform.ScaleX < _max)
        {
            // Zoom in
            ZoomScaleTransform.ScaleX += 0.1;
            ZoomScaleTransform.ScaleY += 0.1;
        }
    }

    else if (delta < 0)
    {
        if (ZoomScaleTransform.ScaleX > _min)
        {
            // Zoom out
            ZoomScaleTransform.ScaleX -= 0.1;
            ZoomScaleTransform.ScaleY -= 0.1;
        }
    }

    if (delta != 0)
    {
        args.PreventDefault();
        e.SetProperty("returnValue", false);
    }
}
Because Silverlight doesn't fire mouse-wheel events, SpinAndZoom responds to the browser's mouse-wheel events, taking advantage of the fact that in Silverlight 2, you can register managed event handlers for unmanaged browser DOM events. Registration takes place in the Page constructor, and the same handler is registered three times to accommodate differences between browsers. Browsers are maddeningly inconsistent in how they report mouse-wheel events, so OnMouseWheelTurned includes some smarts of its own to detect the direction of mouse-wheel movement.

Live Video Overlays the Easy Way
The final example isn't as glitzy as the others, but it demonstrates a simple and inexpensive solution to a problem that could otherwise require extraordinary measures.
I recently met with a Microsoft customer who uses Windows Media Player to stream live events. This customer was interested in using the cross-platform capabilities of Silverlight to expand his customer base to non-Windows users. Almost as an aside, he asked if Windows Media Server would allow him to embed live overlays in the video feed—he wanted something along the lines of the tickers that scroll across the bottom of the screen on television news channels.
I explained to him that with Silverlight, you don't need expensive hardware or software on the server to do video overlays. Instead, you can use Silverlight's networking stack to grab a feed out of band with the video stream, and then use XAML to display the feed on top of a Silverlight MediaElement. The application pictured in Figure 9 shows how.
Figure 9 Scrolling Headlines Positioned over a MediaElement
This application uses a MediaElement to play a WMV (Windows Media Video) file downloaded from the server. Out of band to the video, it uses a WebClient object to fetch a news feed from FeedBurner.com (which has an XML policy file in place permitting cross-domain access) and a SyndicationFeed object to parse the feed and generate a string of news headlines. Then it displays the string in an overlay consisting of a TextBlock and a Rectangle, using an opacity of 0.5 to prevent the overlay from completely obscuring the video underneath. A simple animation scrolls the TextBlock horizontally, and a clipping region clips out all of the text outside the Rectangle (see Figure 10).
Figure 10 Page.xaml
<UserControl x:Class="VideoOverlay.Page"
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
  <Grid x:Name="LayoutRoot" Background="Black">
    <Canvas Width="720" Height="480">
      <MediaElement x:Name="Player" Source="Videos/CRCC Jet Fly.wmv"
        Width="720" Height="480" MediaEnded="MediaElement_MediaEnded" />
      <Canvas Canvas.Top="448" Width="720" Height="32" Opacity="0.5">
        <Canvas.Clip>
          <RectangleGeometry Rect="0,0,720,32" />
        </Canvas.Clip>
        <Rectangle x:Name="Marquee" Width="720" Height="32" Fill="Black" />
        <TextBlock x:Name="Headlines" Canvas.Left="800" Canvas.Top="8"
          Foreground="White" FontSize="14">
          <TextBlock.Resources>
            <Storyboard x:Name="TickerStoryBoard"
              Completed="TickerStoryBoard_Completed">
              <DoubleAnimation x:Name="TickerAnimation"
                Storyboard.TargetName="Headlines"
                Storyboard.TargetProperty="(Canvas.Left)" />
            ...
</UserControl>
Parts of the XAML codebehind appear in Figure 11 (the code download contains the full code). The Page constructor calls WebClient.OpenReadAsync to launch an asynchronous request for the news feed, and the completion event handler copies the headlines from the feed into a TextBlock. Then it calls Storyboard.Begin to begin scrolling the TextBlock and programs a DispatcherTimer to fire a Tick event every five minutes. The Tick event handler submits a fresh request for the news feed, which is ultimately stored in a private field.
Figure 11 Page.xaml.cs
using System;
namespace VideoOverlay
{
    public partial class Page : UserControl
    {
        private const double _offset = 20.0;
        private const double _secondsPerFrame = 10.0;
        private const string _separator = "   ?   ";
        private readonly Uri _uri =
            new Uri("http://feeds.feedburner.com/AbcNews_TopStories");
        private DispatcherTimer _timer = new DispatcherTimer();
        private string _text = null;

        public Page()
        {
            InitializeComponent();

            // Launch an async request for current news headlines
            WebClient wc = new WebClient();
            wc.OpenReadCompleted += new
                OpenReadCompletedEventHandler(OnInitialDownloadCompleted);
            wc.OpenReadAsync(_uri);
        }

        private void OnInitialDownloadCompleted(object sender,
            OpenReadCompletedEventArgs e)
        {
            if (e.Error == null)
            {
                // Convert the news content into a string
                Headlines.Text = GetHeadlinesFromSyndicationStream(e.Result);

                // Begin scrolling headlines
                StartTicker();

                // Start the refresh timer
                _timer.Tick += new EventHandler(OnTimerTick);
                _timer.Interval = new TimeSpan(0, 5, 0); // 5 minutes
                _timer.Start();
            }
        }

        private void OnTimerTick(object sender, EventArgs e)
        {
            // Launch an async request for current news headlines
            WebClient wc = new WebClient();
            wc.OpenReadCompleted += new
                OpenReadCompletedEventHandler(OnRefreshDownloadCompleted);
            wc.OpenReadAsync(_uri);
        }

        private void OnRefreshDownloadCompleted(object sender,
            OpenReadCompletedEventArgs e)
        {
            if (e.Error == null)
            {
                // Convert the news content into a string and store it away
                _text = GetHeadlinesFromSyndicationStream(e.Result);
            }
        }

...
    }
}
Each time the animation that scrolls the headlines ends, a Storyboard.Completed event handler checks that field for a fresh news feed and updates the contents of the TextBlock if new content is available. It then starts the animation running again. Consequently, headlines scroll across the screen in a continuous loop, and the headlines are refreshed every five minutes. Significantly, there is no need for thread synchronization logic or for marshaling calls back to the application's UI thread; this is because both the DispatcherTimer.Tick event handler and the Storyboard.Completed event handler execute on the UI thread.
The downside to this approach is that fetching and refreshing the news feed generates additional network traffic. But that traffic pales in comparison to the video stream itself, and it doesn't require expensive hardware or software on the server to modify live video streams. Moreover, a news feed rendered on the client can be enhanced with additional UX effects, such as mouseovers that display additional information about a headline or hyperlinks that link headlines to news pages. The possibilities are endless when content presented to the user is part of the XAML DOM rather than just bits in a video stream.
Transforms and clipping regions are among the most powerful tools that XAML places at your disposal. Without them, the graphics subsystem of Silverlight would be a lot less extraordinary. I'm receiving lots of queries these days about how to implement effects such as rotating carousels, Rolodex-style paging, and interactive charts. Look for more exciting graphical Silverlight goodies in future installments of Wicked Code.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET. He's also cofounder of Wintellect (www.wintellect.com), a software consulting and education firm that specializes in Microsoft .NET.


Foundations
Unit Testing Workflows And Activities
Matt Milner
Code download available at: Foundations2008_11.exe (168 KB)
Browse the Code Online
Contents
Testing Considerations
Testing Activities
Testing Rules
Testing Runtime Services
Testing Workflows
As an author and instructor, I often get asked questions about how to test Windows Workflow Foundation (Windows WF) components. Part of learning to use a new framework is figuring out how to incorporate the tools and components into your development process. This month I will be discussing some of the challenges and techniques related to testing activities, workflows, and associated components. This is in no way intended to be a best practice guide for unit testing or test-driven development. Rather, it is meant to provide you with the techniques you can use to test your workflows in your preferred development methodology.

Testing Considerations
When writing unit tests, you need to test common paths through your code, but things are never quite that simple. In addition to the "happy path" or the expected execution path, you also must test exception paths. In addition, many items to be tested have dependencies, and it would be unrealistic to test through them all. For example, a business object may rely on a data access layer. In testing, however, rather than relying on the database, it is common to use mock objects—objects that have the same interface as the data access layer but do not access the database at run time.
These same issues are present with workflow, but in some cases the approach is slightly different. In other cases, just the fact that Windows WF is a new technology can cause some developers difficulty in deciding how to test their code.
In addition to typical testing issues, using a framework like Windows WF introduces other complexities. Because Windows WF is based on a runtime that manages the execution of workflows and activities, testing must, in almost all cases, involve the use of the runtime. Many of the classes involved in the runtime execution are sealed or otherwise protected, so mocking those objects becomes nearly impossible. In addition, it would be difficult to ensure that any mock objects you created provided all of the necessary functionality found in the runtime. For those reasons, testing workflows or activities generally requires the runtime.
Using the workflow runtime introduces several interesting challenges. First, because the runtime can use different schedulers, each with its own threading model, unit tests must be written with a particular threading model. The best scheduler for testing may not be the same as the scheduler that best fits the ultimate host for your workflows.
The runtime provides services for developers that are helpful at run time but can make testing more challenging—exception handling, for example. The runtime catches all exceptions and manages them in the workflow. This means testing for expected exceptions will work a bit differently than in standard Microsoft .NET Framework code.
When testing a solution built on Windows WF, there are several components that generally need testing: activities, workflows, rules and custom runtime services. Throughout the rest of this article, I will cover the common issues developers run into when testing these components and propose techniques for addressing them.

Testing Activities
When testing activities, it is important to be able to do so in isolation, providing inputs and testing outputs. Rather than testing activities in the context of a workflow, activities should be tested as individual components. Fortunately, in Windows WF, all workflows are simply activities. And that means the runtime can execute any activity as a workflow, even something as simple as an e-mail activity.
Before executing the activity, the runtime must be set up and the appropriate scheduler and other runtime services added to it. Generally, when testing activities, the ManualWorkflowSchedulerService is easiest to use, as it allows the activity to execute on the same thread as the unit test and provides the most control over the execution. The scheduler does not do any work in the workflow or activities until the host, in this case the test, provides the thread to do the work.
When testing workflows, I typically use test setup and teardown methods to initialize the runtime and gracefully shut it down. In the test methods, I can focus on executing a particular activity test knowing that I have a clean runtime in which to execute the test. Figure 1 shows code for a simple activity test that uses the functionality in Windows WF of passing parameters to a workflow to initialize it.
Figure 1 Executing an Activity as Workflow
Dictionary<string, object> results = null;
Exception ex = null;

runtime.WorkflowCompleted += delegate(
  object sender, WorkflowCompletedEventArgs wce) {
  results = wce.OutputParameters; };

Dictionary<string, object> wfParams = new Dictionary<string, object>();
wfParams.Add("To", "mmnet30@microsoft.com");
wfParams.Add("From", "donotreply@example.org");
wfParams.Add("Subject", "Unit testing");

WorkflowInstance instance = runtime.CreateWorkflow(
  typeof(WFComponents.SendMailActivity), wfParams);
instance.Start();

ManualWorkflowSchedulerService man =
  runtime.GetService<ManualWorkflowSchedulerService>();
man.RunWorkflow(instance.InstanceId);

//the workflow is done, now we can test the outputs
Assert.IsNotNull(results, "No results found");
Assert.AreEqual<string>(wfParams["To"].ToString(),
  results["To"].ToString(), "To address was changed");
There are several things to notice in this code. First, the SendMailActivity is used as the definition of the workflow and executed independent of any containing workflow. Parameters are passed to the activity using standard Windows WF techniques where a dictionary of values is passed to the runtime and mapped to the properties on the activity. The manual scheduler is employed to execute the workflow, and, therefore, the RunWorkflow method is used when the activity is ready to be executed. Once the RunWorkflow method has completed, the simple activity has finished and the output parameters can be extracted and assertions made.
Notice that the output parameters are collected during the WorkflowCompleted event and the reference mapped to a local variable that can then be inspected. Because the manual scheduler is being used, the WorkflowCompleted event execution happens on the same thread as all other work in this test.
Other schedulers can be used during testing, but that would require more complexity in the test code. For instance, if the default workflow scheduler were used, you would need a wait handle of some sort to block execution until the workflow completed. You can see an example of this type of waiting in Visual Studio by creating a new sequential workflow console application and examining the program.cs file.
The first test takes care of the normal execution, but what about exception cases? Most unit testing frameworks have the ability to declare expected exceptions to simplify test development. For example, this code is an example of how the unit testing in Visual Studio 2008 allows you to declare an expected exception on a test method:
[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void MailWithInvalidFrom(){...}
The problem with using this method when testing in Windows WF is that the runtime catches exceptions and, rather than throwing them in the host, raises an event when it terminates the workflow, passing the exception as part of the event arguments. This provides two options for handling exceptions: do not use the built-in behavior of the testing framework, or handle the terminated event and re-throw the exception.
You might think that throwing the exception from the event handler would work, but unfortunately it does not as the runtime catches the re-thrown exception. Figure 2 shows another test of the SendMailActivity, this time failing to pass a From address, which should cause an ArgumentNullException in the code.
Figure 2 Dealing with Exceptions
[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void MailWithInvalidFrom() {
  Exception ex = null;

  runtime.WorkflowTerminated += delegate(
    object sender, WorkflowTerminatedEventArgs wte) {
    ex = wte.Exception; };

  Dictionary<string, object> wfParams =
    new Dictionary<string, object>();
  wfParams.Add("To", "mmnet30@microsoft.com");
  //wfParams.Add("From", "donotreply@example.org");
  wfParams.Add("Subject", "Unit testing exception");

  WorkflowInstance instance = runtime.CreateWorkflow(
  typeof(WFComponents.SendMailActivity), wfParams);

  instance.Start();

  ManualWorkflowSchedulerService man =
    runtime.GetService<ManualWorkflowSchedulerService>();
  man.RunWorkflow(instance.InstanceId);

  //exception should be thrown by now
  if (ex != null)
  throw (ex);
  else
  Assert.Fail("Exception should have been thrown");
}
As you can see, the exception can be pulled from the arguments in the WorkflowTerminated event. But once that has happened, it is necessary to throw the exception in order to use the ExpectedException behavior. At this point you might decide it is just as easy to use an Assert and make sure the exception is of the correct type rather than throwing the exception and using the ExpectedException attribute.
Up to this point, you probably noticed that the testing code has not tested the actual output of sending the e-mail. In the case of e-mail, it is easy to configure the system.net/mailSettings configuration in the application configuration file to make sure messages are written to the file system. But then those messages must be opened and read to validate the output. A better way to test this functionality would be with a mock object that abstracts away the e-mail process.
In Windows WF, activities can either do work directly or use objects that are loaded into the runtime known as runtime services. One of the benefits of using runtime services to handle work is that the host process is in charge of adding the services into the runtime. If the activity is going to be used in an ASP.NET application, the code might be different than if the activity is being used in a Windows service. The activity simply queries the runtime for an object of a particular type then calls methods on it.
Using this model works very well for testing, as the runtime service can be replaced during tests with mock objects to help the testing effort. This might seem like a lot of extra work in your activities and a stiff requirement that all hosts must add a runtime service to support your activity. However, your activity can be written in such a way that if a runtime service is present, it uses it, and if not, the activity provides its own implementation.
Here is the execute method of the SendMailActivity showing how the execution can be different depending on whether a runtime service is present. If the service is found, the message is sent via the service; otherwise, the message is sent directly using the System.Net.Mail namespace:
IEmailService mailSvc =
  executionContext.GetService<IEmailService>();

MailMessage msg = new MailMessage(From, To);
msg.Subject = Subject;

//use the mail service if it is present
if (mailSvc != null) {
  mailSvc.SendMessage(msg);
}
else {
  SmtpClient client = new SmtpClient();
  client.Send(msg);
}
Notice that the activity is using an interface instead of a concrete class, which provides a key abstraction point. The activity is not dependent on a specific implementation, rather it accepts the implementation of the service that is added by the runtime. The test then is able to use a mock implementation of the service to aid in testing as shown in Figure 3.
Figure 3 Testing Activities with a Mock Runtime Service
[TestMethod]
public void TestEmailWithMock() {
  MockEmailService mock = new MockEmailService();
  runtime.AddService(mock);

  ...

  //the workflow is done, now we can test the outputs
  Assert.IsNotNull(results, "No results found");

  MockEmailService email =
    runtime.GetService<MockEmailService>();
  Assert.IsNotNull(email,
    "No email service found in the runtime");

  Assert.AreEqual<string>(wfParams["To"].ToString(),
    email.Message.To.ToString(),
    "To address was changed");
}
Now the e-mail does not actually have to be sent as part of the test, and the outcome of the test is easier to validate using the implementation that was added. This method of abstraction works well for general hosting flexibility as well as testing.

Testing Rules
A critical component of many workflow solutions, business rules provide additional challenges to testing. Rulesets in Windows WF are not all that challenging to execute outside the context of a workflow, but a ruleset is defined based on a particular type, and that type is usually the workflow class. In general, rules need an object or set of objects as inputs that they can be evaluated against and take actions on. In the case of Windows WF, the workflow itself is usually the root object, and any other objects with which the rules interact are simply properties or activities in the workflow.
When you are executing workflows, the Policy activity is most often used to execute rules. In a unit test scenario, rules are better tested directly without an activity. To execute a ruleset, several classes from the System.Workflow.Activities.Rules namespace can be used. In addition, the RuleSet class can be deserialized from a stream using the WorkflowMarkupSerializer class so the representation of your rules that you use at runtime can be used in your test environment.
Figure 4 shows the basic code used to execute a ruleset in a unit test. Once the ruleset has finished executing, the objects used during the rule execution can be used in assertions to validate expected behavior. The important thing to notice first in this example is that the rules are being executed against a customer object instead of a workflow type. This is very intentional for several reasons.
Figure 4 Executing a Ruleset during a Test
//initialize test object
Customer cust = new Customer();
cust.Level = CustomerLevel.Gold;
cust.CurrentAnnualPurchases = 1578;

//create rule execution objects
RuleValidation val =
  new RuleValidation(typeof(Customer), null);
RuleExecution exec = new RuleExecution(val, cust);
RuleSet ruleset = null;

//get rules from file / deserialize
WorkflowMarkupSerializer serializer =
  new WorkflowMarkupSerializer();

string rulePath = Path.Combine(
  testContextInstance.TestDeploymentDir, "Customer.rules");
RuleDefinitions defs =
  (RuleDefinitions)serializer.Deserialize(
  XmlReader.Create(rulePath));
ruleset = defs.RuleSets[0];

//execute ruleset
ruleset.Execute(exec);

//test the outcome
Assert.AreEqual<BLL.CustomerLevel>(cust.Level,
  CustomerLevel.Platinum);
The default mode of creating rulesets using the Policy activity means that the policies are created based on the workflow type and are embedded as a resource in the assembly. This causes problems not just for testing—it can also be a challenge to flexibility at run time. For testing it provides a challenge, because in order to test the ruleset an instance of the workflow must be created and initialized. Creating rules against other types, such as business objects, makes testing much easier and can make using the rules from within your workflow easier. You can find several examples of working with rules, including creating custom activities, on the Windows WF MSDN site.
Testing the rules involves creating and initializing the object that acts as the input for the rules and initializing the ruleset and execution objects, which includes the rule validator and executor. The code shown is mostly boilerplate and can be used in custom activities that are built to execute rules on custom objects instead of workflows.
Sometimes this method of isolating the inputs for the rules into a separate type is not feasible, as the rules are dependent on properties of various activities in the workflow. In those cases, an instance of the workflow must be created using the CreateWorkflow method of the runtime and the properties of the activities initialized. Properties on the workflow can easily be initialized by passing parameters when creating it, but initializing the activities can be a bit more challenging. If the properties on the activities are dependency properties, then they can simply be bound to properties on the workflow and therefore initialized at time of creation.
If, however, the properties on the activities are standard .NET properties, a different approach is required. A simple approach is to put code in the Initialize method of the workflow and find all of the activity instances in the hierarchy and set their values. Note that you cannot do this work from the unit test itself because, once the workflow has been created, the host code does not have direct access to the running instance state.
The final wrinkle comes when using activities as inputs to your rules when the activities themselves are dynamically created, as in a While activity or a state machine. For more information on the issues related to iterating activities, see the June 2007 Foundations column, "ActivityExecutionContext in Workflows". In these scenarios, the only way to have the activities correctly initialized for testing is to execute a workflow and have the composite activity create the activity clones. I am not going to spend time discussing this technique in detail because most developers shy away from writing rules in this scenario, as the rules themselves get too complex.

Testing Runtime Services
Runtime services refer to any .NET object added to the workflow runtime. There are services that ship with the Framework providing persistence, tracking, scheduling and other well-known services with which the runtime is familiar. These are services that are part of the framework and should not be the focus of your testing. However, when developing custom solutions, two kinds of runtime services might need testing: custom services that provide known functionality to the runtime and custom services that interact with activities or the runtime.
In the case of services that provide known functionality, such as a custom persistence service, the only real way to test these services is by executing them in the runtime. To manage the execution and control the test, the best method is to use a workflow as a driver. Using a workflow allows for controlling conditions, such as workflow idling, suspensions, and terminations, that might cause your service to be invoked.
When it comes to other services to be tested, the requirements for testing depend on the implementation of the service. For example, some services interact with the runtime via the events that it raises. Other services interact more directly with information about a workflow instance, such as requiring the instance identifier or using the current WorkBatch when called by an activity. In these scenarios, the service will be best tested in the context of a driver workflow. The methods shown at the beginning of this article can be used to drive the test.
Some objects only exist in the runtime to provide services to activities and do not rely on any other services. If this is the case, these services can generally be tested as standard .NET components. While there are no requirements that must be met from the perspective of the workflow runtime, the tests may still use mock objects for other services or dependent objects. Optionally, a single activity can be used as a driver, as shown earlier in this column, where the activity fully exercises the service ensuring that the tested functionality works when executed in the runtime. This last option might seem like extra work, but it also provides a bit more confidence that your code will execute correctly at run time.

Testing Workflows
One of the biggest challenges comes when developers attempt to unit test entire workflows. Because workflows often model long running processes that interact with many different services and applications, the complexity in testing quickly expands. For example, a given workflow may call several services using Windows Communication Foundation (WCF) and aggregate the results together as part of its logic.
It is possible to handle these interactions by injecting mock objects for testing. In fact, with the WCF activities in Windows WF, the ability to inject a testable service endpoint is simplified. The Send activity attempts to use the ChannelManagerService class to resolve endpoints. A test class can easily add named endpoints that match those expected in the workflow, changing the binding and address to point to a local service implementation. Using this indirection, a workflow that invokes several services can be tested in many cases. Please see the August 2008 issue of MSDN Magazine for more information on using the ChannelManagerService class.
In summary, it is not really a question of whether you need to test your workflow, rather it is a question of how you will test them. In many development teams, the testing of the workflow is not considered a unit testing task, rather it is an integration test that would take place after unit testing was complete and the various components are known to be in good working order. If you choose to use unit tests on your workflows or a driver to other automated testing, you can use the information presented in this article to help execute your tests.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Workflow, BizTalk Server, and Windows Communication Foundation. Matt lives in Minnesota with his wife, Kristen, and his two sons. Contact Matt via his blog at pluralsight.com/community/blogs/matt.


Team System
Team Build 2008
Brian A. Randell
Code download available at: TeamSystem2008_11.exe (165 KB)
Browse the Code Online
Contents
About Team Build
Creating a Team Build
Creating a Build Agent
Running a Team Build
Programming Team Build
Team Foundation Server (TFS) is about teams. Sure, a team of one can use it, but TFS is really all about collaboration with other humans on your development projects.
Build automation is not F5. Build automation is about collecting, assembling, validating, and auditing. The idea is that you will collect all of the artifacts that comprise your solution and assemble said artifacts based upon a well-defined integration plan. Compilation is a typical task defined in the assembling phase. You use various types of automated tests, often know as build verification tests (BVTs) or smoke tests, to then validate your team's collective work.
Finally, you'll perform some type of audit to assess the overall quality of the build. The audit helps provide data to help you answer the tough questions. For example, do the bits as of this build create a product? Is your product of high quality? Do you have any regressions? Are you on schedule? The value of build auditing is making it trivial to take the pulse of your team.
In this column I will introduce you to Visual Studio 2008 Team Foundation Server Build (Team Build) and walk you through the process of creating and running a team build. I'll also demonstrate the managed API now available that lets you program against Team Build 2008.

About Team Build
Team Build is a core feature of TFS 2008. Microsoft designed it to be an industrial-strength build automation tool. You could say it provides the F5 experience for your team. In a typical environment, you'll install Team Build on its own dedicated server (although if you have a very small team or a tight budget, you can install Team Build on your TFS box).
Unlike other portions of TFS, the 2008 release of Team Build was a major update. In the 2005 release, Microsoft created a good build automation product. Using MS-Build as its engine, Team Build 2005 supported many of the core automation features you'd expect: version control, compilation, multiple build types, unit tests, static analysis, and code coverage.
In addition, it provided excellent reporting features including build reports and data updates to the TFS data warehouse to support historical trend analysis. The 2005 release of Team Build also supported multiple-build machines, build notifications, and the ability to start a build from the command line or through the Team Explorer user interface.
For Team Build 2008, Microsoft did more than just spit and polish. Major enhancements include continuous integration support, build queuing, scheduled builds through Team Explorer, and a managed object model. In addition, Microsoft separated build agent definition from build definition, created a build definition editing user interface, and added better build management tools.
Combined with the solid Team Build 2005 feature set, Team Build 2008 presents a compelling build automation solution. In addition, because Microsoft continues to support backward compatibility, you can upgrade your TFS installation and your Team Build servers even if your development team is still using Visual Studio 2005.

Creating a Team Build
To use Team Build you naturally need a TFS installation, either a single server or dual server configuration. You'll need to have installed Team Build and properly configured your build server. The details are covered in the recently updated Team Foundation Installation Guide. Finally, you'll need a team project with some code checked into the version control repository.
In Visual Studio 2008 with a Team Foundation Server client installed, open the Team Explorer window with your team project loaded. In this example, let's assume you've created a team project called MSDNMag using the MSF-Agile template. (If you want to follow along, download the code from this column as well as the Team System 2008 TFS and Team Suite Virtual PC image.) This example uses a custom workspace called MSDNMag with the trunk mapped to C:\Work in the Virtual PC image.
With all of that out of the way, you can expand the MSDNMag Team Project node in the Team Explorer window and locate the Builds folder. If you right-click on the Builds folder, you'll be presented with a number of options. Of concern to you at this point is the New Build Definition command. Once selected, the Build Definition dialog appears (see Figure 1).
Figure 1 A Blank Build Definition Dialog
Unlike the wizard in Team Build 2005, you can return to this dialog when you need to edit a build definition. First you provide a build name such as SimpleBuildExplorer-OnDemand. In addition, you can keep the build offline by checking the Disable this build definition option.
When you create a new build, it's mandatory that you provide a build definition name, a Team Build project file, and that you specify the build agent and drop location. That said, the second option, Workspace, is critical in my opinion.
In Team Build 2005, the Workspace option was something you just had to know about. The implementation details have changed, but the purpose is the same: to limit what artifacts Team Build downloads during the get sources step of the build process. In Team Build 2005, you had to modify a file called WorkspaceMapping.xml. In Team Build 2008, you use a GUI similar to that found in the Workspace mapping dialog for version control (see Figure 2).
Figure 2 The Build Definition Dialog’s Workspace Page
Workspace lets you define what sections of your source code repository you want used as part of the build. By default, Team Build maps the trunk of your Team Project to the source download location used by the build (something that you can control when associating the build agent that runs the build with the build definition). To be clear, if you don't adjust this setting, every time you execute a build, Team Build will do a get on the trunk, pulling down the latest of everything. Not a big deal when you're doing demos or testing, but once you've got six months of work in your team project and multiple branches, it's not the best way to run a build.
Even though the demo scenario here only has one Visual Studio solution, you should still adjust the mapping. When you click the first item under the Source Control Folder column, you'll have the option of either typing in a restricted path or using the Browse for Folder dialog. For this example, I set the Source Control Folder to $/MSDNMag/Main/src/TeamSystem/C08.
Just as in the dialog used for version control workspace mapping, you can define more than one folder mapping. The variable $(SourceDir) is an environment variable that Team Build expands to the build directory specified for the build agent in the build agent properties dialog. In general, you will want use $(SourceDir) so that paths are relative. However, Team Build does not force you to use it. For this example, I leave it as is.
Finally, if you have an existing workspace that has all of the mappings configured how you want, you can use the Copy Existing Workspace option to copy the version control folder part of the mapping, saving you time.
Once you've defined your workspace settings, you need to define a build project file. When you click on the Project File option, you have the option of providing the path to an existing TFSBuild.proj file that is already under version control. TFSBuild.proj is an XML file that in the 2005 release contained all the build configuration data and was the file you customized to enhance the actual build process. Team Build 2008 continues to use this file for build customization, but most of the data related to the build definition is stored in the TFS databases. If you don't have a file already, you click the Create button and build one using a simple wizard.
One big change in Team Build 2008 is that you can now specify where you'd like to store your TFSBuild.proj file. In Team Build 2005, Microsoft hardcoded the path to a specific location in the version control repository. By default, the Team Build 2008 dialog specifies a Team Build 2005-compatible path in the form of %TeamProjectName%/TeamBuildTypes/%BuildDefinitionName%.
For this current example, you will see $/MSDNMag/TeamBuildTypes/SimpleBuildExplorer-OnDemand for the proposed version control folder. The major benefit of being able to change the path is that you can now branch your TFSBuild.proj file with other artifacts. For this example, accept the default path and click the Create button.
When you click the Create button, Team Build presents you with a wizard. The wizard is one of the few UI remnants from the Team Build 2005 release. The first page asks you to select the Visual Studio solutions you'd like to build. The wizard filters the list presented to only those solutions available relative to your workspace mapping. If you don't change the default workspace mapping, you'd see all the solutions in your Team Project. Select the single solution provided and click Next.
The second page of the wizard allows you to choose what type of Visual Studio configurations you'd like built. While the wizard only exposes Debug and Release for configuration, you can type in custom configurations. The same rule applies to Platform. Accept the defaults and click Next.
The final page of the wizard lets you control test and static analysis execution. VSTS needs to be installed on the build agent computer in order for test execution and static analysis to work. As was possible with the Team Build 2005 release, you can choose a test metadata file and from this select any test lists that support unattended execution. However, you can alternately choose to have Team Build automatically scan your output assemblies for MSTest compatible tests and run them without the need for a test list. You can choose to have Team Build execute static analysis if you've enabled it in one or more of your Visual Studio projects. The sample solution for this column doesn't have any tests or static analysis, so click Finish to complete the wizard and create the TFSBuild.proj file.
At this point, you only have to complete the build defaults. First, let's look at the retention policy and trigger settings. A huge improvement over the Team Build 2005 release, Team Build 2008 now lets you configure per build definition retention policies (see Figure 3). Based upon a particular build's outcome, you can control how many build results Team Build keeps.
Figure 3 Controlling a Build Definition Retention Policy
Once a build completes, you can mark a build result so that Team Build keeps it regardless of the retention policy. For this example, note that you should leave the retention policy set to the defaults.
If you click the Trigger option in Team Build 2008, it works just like Team Build 2005—it only runs builds when you ask. However, you'll note that now Team Build also supports continuous integration (CI). (For more information on CI, see the March 2008 article "Redefine Your Build Process with Continuous Integration".) This means a build is queued whenever someone checks in a change to a file that is mapped by a build definition's workspace mappings. The dialog exposes this with the Build each check-in (more builds) option.
A second option, Accumulate check-ins until the prior build finishes (fewer builds), provides a delayed form of CI. Finally, the last option lets you configure a scheduled build (see Figure 4).
Figure 4 Configuring Team Build to Execute Builds on a Schedule
You should leave the trigger set to the default of Check-ins do not trigger a new build, and select the Build Defaults option. Use the build defaults to specify the build agent to use and the drop location for the build output. You specify the drop location using a UNC path. If you're using the Microsoft provided Virtual PC image, you can enter \\localhost\Drops\MSDNMag\SimpleBuildExplorer-OnDemand. Only the share needs to exist. Team Build will create additional subfolders as necessary. Team Build will create a folder under this path using the build number.
When creating the share, you need to ensure you assign proper permissions. At a minimum, the Team Build and TFS Service accounts need Full Control rights to the share. In addition, you need to give Write permissions to anyone who is going to publish test results related to the build. You should give Read permissions to those people who need to pull down completed builds.
Assuming you've got the share configured, you need to specify a build agent. If you've already defined a build agent, you can choose it from the combobox. If not, you can click the New button, which opens the standard Build Agent Properties dialog. You'll need to define a new agent for this example, so click New.

Creating a Build Agent
Unlike in the Team Build 2005 release, Team Build 2008 build agents are independent of build definitions. Once you've defined an agent, you can easily use it again. The Build Agent Properties dialog lets you define all of the critical properties related to the agent. You first provide a display name and optional description. You then provide the computer name in the form of a NETBIOS or DNS name. The display name and computer name can be completely different. If you're using SSL, you can specify that option.
A critical setting is the working directory. By default, it uses a path in the form of $(Temp)\$(BuildDefinitionPath). While this works for many builds, it can cause problems due to the dreaded MAX_PATH issue. $Temp on a Windows Server 2003 machine will be %SystemDrive%\Documents and Settings\<build agent service account>\Local Settings\Temp. That's too many characters to be giving up. I generally recommend people change this to something shorter, especially if building Database Professional builds. For this example, change the working directory to C:\Builds\$(BuildDefinitionPath).
Finally, you can control the agent's status. This is particularly useful for servicing. In addition, TFS will change an agent's status to unreachable if it can't contact a build agent, thus preventing use of the agent until an administrator changes the setting. Leave the setting set to enabled and click OK to close the completed Build Agent Properties dialog (see Figure 5). Back in the Build Definition dialog, all of the warning triangles will now be gone and you can click OK to complete defining the build.
Figure 5 Completed Build Agent Properties Dialog
In the Team Explorer window, under the Builds folder, you'll now see your build definition. If you right-click on it, you can see there is now an Edit Build Definition option that returns you to the same Build Definition dialog you just used. In addition, you can queue a new build using the Queue New Build option.
Unlike Team Build 2005, Team Build 2008 now supports queuing of builds—a necessary feature if Microsoft was going to properly support continuous integration. When you select this option, the Queue Build dialog opens (see Figure 6) and you can adjust a few options. First, you can easily change the default build agent to an alternative if you would like. Second, you can adjust the drop location. In addition, you can have a particular build "jump the queue" by adjusting the Priority in queue option. Finally, you can pass MSBuild command-line arguments if the user has the Administer Builds permission.
Figure 6 Queue Build Dialog

Running a Team Build
Click Queue to start the build. When you do, the updated Build Explorer window will open. It has two main tabs. The queued tab by default shows you all builds that are queued regardless of agent or status. You'll find comboboxes that you can use to adjust which builds the Build Explorer shows based upon build definition, the build's status, or the build agent.
The completed tab displays the completed builds and supports the same type of filtering. When you start your build, you can double-click on the build to view the live build report. Team Build updates this report in real time as it progress through the build process.
When you run a team build, a number of actions occur. Once TFS starts the build from the queue, the build agent is prepared to run the build. TFS creates a build number and the build agent synchronizes sources with the location specified in the build definition. The agent then compiles the code and runs code analysis if required. The build agent then runs tests. Then the agent runs a task that updates any associated work items with the build number.
For example, the default Task work item from the MSF-Agile template has an Integration Build field that Team Build updates. The agent calculates code coverage (if tests are configured to run as part of the build and code coverage is enabled) and determines all of the changesets between the last good build (compilation was successful) and the current build. It copies the files up to the drop share, publishes the build results to TFS, and TFS fires a build completion event. See my May 2008 column for details about TFS events.
There are two important changes that Microsoft made in Team Build 2008 related to tests and work items. The first change is that a build that has one or more failing tests does not fail the build. Instead, a build with one or more failed tests will show as Partially Completed, not Failed, as in Team Build 2005. Second, in Team Build 2005, the first time you ran a build, Team Build would list all of the work items and changesets related to the items built on the first build report since the beginning of the team project. Team Build 2008 only starts showing and updating work items after a build's first successful run. In my next column, I'll cover how to adjust these behaviors through build customization.

Programming Team Build
One of the great enhancements made in Team Build 2008 is the addition of a managed API. The Microsoft.TeamFoundation.Build.Client.dll is the primary assembly you will use. The sample application for this column references this assembly as well as the Microsoft.TeamFoundation.Client.dll and the Microsoft.TeamFoundation.VersionControl.Client.dll assemblies.
The sample application is a simple Windows Forms application that exercises some of the new API. The application connects to TFS and enumerates the list of all team projects, loading them into a tree view control. Under each team project, it adds Build Agents and Build Definitions nodes (see Figure 7).
Figure 7 Sample Application
When you expand a node, code in the TreeNode's BeforeExpand event runs and enumerates the available build agents, build definitions, or completed builds depending upon the node you've expanded. In addition, the main form hosts a Property Grid control. When you select a node that has a non-null tag value, the application loads the reference object into the Property Grid's SelectedObject property. Figure 7 displays the application with the MSDNMag team project expanded and the Main Build Agent selected.
Most of the application is simple enumeration code. However, it has one feature that Microsoft did not surface in Team Explorer. When you define a build agent, TFS scopes the build agent to that particular team project. I've found, however, that many organizations only have one or two build agents. That means they end up having to recreate the build agent multiple times. Using the sample application, if you right-click on a build agent instance, you'll be presented with a context menu and a Copy Agent To command (see Figure 8).
Figure 8 Copy Agent To Command
Choosing this option opens a dialog that allows you to chose the Team Project to which you wish to copy the build agent definition. All of the interesting code is in the CopyAgentToToolStripMenuItem_Click event handler shown in Figure 9.
Figure 9 CopyAgentToToolStripMenuItem_Click Method
Private Sub CopyAgentToToolStripMenuItem_Click( _
  ByVal sender As System.Object, _
  ByVal e As System.EventArgs) _
  Handles CopyAgentToToolStripMenuItem.Click

  Dim current As TreeNode = tv.SelectedNode
  Dim ba As IBuildAgent = _
    TryCast(current.Tag, IBuildAgent)

  If ba Is Nothing Then Exit Sub

  Dim parentProject As String = ba.TeamProject

  Using frm As New frmPickProject
    frm.cboProjects.DataSource = _
      (From item In teamProjectNames _
       Where item <> parentProject _
       Select item).ToArray()

    If frm.ShowDialog(Me) = Windows.Forms.DialogResult.OK Then
      Dim teamProjectName As String = _
        frm.cboProjects.SelectedItem.ToString()

      Dim existingAgent = _
        (From item As IBuildAgent In _
         m_bs.QueryBuildAgents(teamProjectName) _
         Where item.Name = ba.Name).SingleOrDefault

      If existingAgent Is Nothing Then
        Dim newBuildAgent As IBuildAgent = _
        ba.CopyTo(teamProjectName)
        newBuildAgent.Save()

        MessageBox.Show(newBuildAgent.Name & " copied.")
      Else
        MessageBox.Show(existingAgent.Name & " already exists.")
      End If
    End If
  End Using
End Sub
Assuming the selected node has a tag set to an IBuildAgent reference, the code shows the frmPickProject form. This form contains a combobox that displays all of the available team projects except the current build agent's team project.
The code populates the data source of the combobox by executing a LINQ query against the string array of all team project names gathered when you expand the root of the tree view for the first time. Assuming you select a team project and click OK on the project selection form, the code runs a LINQ query to verify that the target team project does not already have a build agent with the same name. If it finds one, it displays a message. However, if it doesn't find one, it copies the build agent definition.
Of particular interest is that Microsoft provides a CopyTo method. When I first worked out this code, I had planned to do a per-property copy against a new IBuildAgent instance. It looks like Microsoft got the API in but not added to the user interface.
I hope you see that Team Build 2008 is an industrial-strength build product. If you're using Team Build 2005, there's a ton of value in upgrading to Team Build 2008. In my next column, I'll dig into build customization and how you can have your builds not only compile a solution but deploy to test environments and more.
I would like to thank Ken Getz and Martin Woodward for their help with this column.

Send your questions and comments for Brian to mmvsts@microsoft.com.

Brian A. Randell is a Senior Consultant with MCW Technologies LLC. He spends his time speaking, teaching, and writing about Microsoft technologies. Brian is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


{ End Bracket }
Engineers Who Write
Richard Ward

See Richard Ward's "Behind the Code" interview at channel9.msdn.com/Shows/Behind_The_Code.

I was somewhat amused when the MSDN Magazine staff asked me to write a column. There was an initial trepidation, as if engineers were not expected to write and raising such a request was inviting some sort of surly response. This happens more than it probably should, but it indicates a more general issue.
Engineers—and indeed everyone involved in creating software— need to be able to communicate verbally, and that doesn't mean by speaking. Oral communication is great; having a small set of people hashing out details on a whiteboard or over lunch is an invaluable part of completing the design. The rapid interaction allows everyone to bounce ideas around and get immediate feedback. But it is, by its nature, imprecise. The understanding developed during the conversation and the shared context built by the participants rarely extends past the conversation. Only what is written down can be used by people outside the conversation.
Writing down ideas and concepts is essential to the development process. While there are many schools of thought on what constitutes the right level of design for a software product, ultimately some aspects of the design must be written down. And much like the notion that no backup product is good unless you verify its restore abilities, whatever is written down must also be readable by the intended audience. If someone reads the document and can't understand the goals, or worse yet, gets a completely different understanding, then the process has failed.
There are a lot of opinions about what should be used when writing engineering documents. I'm not going to stir up that particular debate since I'm more of a use-the-right-tool-for-the-job sort. These can range from the highly structured and precise to a less formal, natural language. A precise model helps, for example, when specifying the interactions between two instances, such as a network protocol. These forms can express the how of a design very effectively because they are, in a sense, only one step removed from the actual code expressing that design. They do not, however, explain the why of a design.
Natural language is inherently less precise, but it is far richer at expressing concepts outside of the actual design. In a building analogy, the architect may have a design that shows exactly how the atrium is framed and supported, the lighting characteristics, and many other subsystems and details; these all exist in the plans for each specialty. But the architect is equally as likely to have a statement about the purpose of the atrium to draw people into the building, to carry them up stairs, past windows, and many other details. This sort of purpose is best expressed in a non-technical language so that the reader can build a private understanding, one that will serve as the foundation for understanding the rest of the design.
This becomes much clearer as broader audiences for these documents are considered. While the engineering staff might appreciate a precise document that spells out exactly how a component should behave, the rest of the team has different needs. As the design trickles out from the original conception to the organization as a whole, the requirements for communicating with each new audience become broader. It is rarely necessary to take a low-level design document to a customer, but it is often helpful to have the requirements in a form that is both understandable to a diverse audience and written in a way that can appeal to the audience at many levels.
Recently I was asked to review a design document from another team, a proposed change that would affect a number of components. The document was incredibly detailed, specifying virtually every changed function, the memory footprint, load order—all carefully researched. One thing that was missing, however, was a statement of why this change was being made. While all the provided data looked good, there was no framework in which it could be evaluated. The engineer on the project was so deep in the work that he assumed that all the readers would know why the changes were being proposed. This was a document that would benefit greatly from an introduction (in a natural language) that set the picture for the changes being proposed (in a highly structured language).
Writing is not something to be viewed with dark suspicion by the engineers. Writing is an important part of the job: you need to be able to communicate information to others in order to create complete products. While some forms may be more comfortable than others, the real requirement is that the reader receives and understands the message. After all, if you can't make yourself understood in something as soft as a natural language, what chance is there that the completely precise and unforgiving computer is going to give you the results you want?

Richard Ward is a Distinguished Engineer at Microsoft and runs the Core Architecture team as part of the Core Operating Systems Division where he often writes long documents.