Introducing the ValidationAttribute Test Harness on Codeplex

***************************

*** Update: for the latest on how to use the ValidationAttribute Test Harness, check out my latest post in this series...

***************************

This article is an introduction to the ValidationAttribute Test Harness project on Codeplex. This project exists to fill a testing void and maintain quality on ValidationAttributes. Our of the box – the ValidationAttribute Test Harness monitors the declaration of error messages and validates their runtime state. This article briefly discusses the nature of ValidationAttributes, the testing issues around the resource approach and how this new CodePlex project addresses the shortcomings.

The standard thinking around testing ValidationAttribute derived classes is that these are not individually testable – but rather relegated to automated ui testing or exploratory testing. While a certain portion of that is true – we cannot fully recreate the framework that interacts with these properties - we can provide a contextual harness that allows us to test the declarations to ensure that the declarations map to their intended targets (and that intended targets are who they say they are).

Purpose 1: Verifying Properly Localized ValidationAttributes at their Attribution Site

When looking at the ValidationAttribute, you quickly notice three properties that give access to declaring the error message raised when the rule is violated. These three are:

• string ErrorMessage
• string ErrorMessageResourceName
• Type ErrorMessageResourceType

It becomes apparent that there are quality issues, especially when it comes to localization.

1. Setting a string on ErrorMessage means the error message is not localizable.
2. The ErrorMessageResourceName is meant to point to a static public property on a resource class – but this is a magic string.
3. Both the Type and Name must be set for resourced error messages to be identified.
4. There are a number of combination-style errors that can arise.
5. The errors that are thrown are obscure and difficult to locate – with details buried deep in the exceptions.

We need a mechanism to test for these scenarios and provide a list of readable errors that lead us directly to the source and allow us to triage them quickly.

Purpose 2: Verifying Custom Attribute Annotations

Custom-defined Attributes, aside from the standard set, are as wide open as we like them to be. They are especially useful when declaring relationships between data. The ASPNET MVC framework generates one out of the box in it’s application template called PropertiesMustMatchAttribute. The attribute is placed at the class level where it declares two properties (via strings) to declare that the two properties are equal which is useful on the model when resetting a password.

This has all the same problems we see with magic strings -

1. What if the declaration is wrong
2. What if one of the property names change. The stringified name won’t be changed with it automatically.
3. What if only one or none of the property names is declared?

Waiting for runtime to discover these kinds of errors is unacceptable.

Using the ValidationAttribute Test Harness to Raise the Quality Bar

Out of the box, the ValidationAttribute Test Harness will take a look at all the ValidationAttribute attributions in your project and verify the configuration of the Error Message related attributes. It will also pull together error information into one concise list that can be easily triaged.

   1: var analyzer = new ValidationAssemblyAnalyzer(typeof(ChangePasswordModel).Assembly, true);

   2: Assert.That(analyzer.ValidationErrorInfoList.Count, Is.EqualTo(0), analyzer.CompileErrorMessage());

The analyzer can also take lists of custom validators so we can provide customized tests based on the attribute type like so:

   1: var analyzer = new ValidationAssemblyAnalyzer(typeof (ResourcedValidationAssemblyAnalyzerTests).Assembly, true, 

   2:                           null,

   3:                           new[] {new PropertiesMustMatchValidator()});

   4:  

   5: Assert.That(analyzer.ValidationErrorInfoList.Count, Is.EqualTo(0), analyzer.CompileErrorMessage());

… this is a great place for MEF, but I’ve held off to avoid dependencies on this initial release. If people just copy the code into their projects, great. If they tend to grab the assembly as a dependency in their unit test project, then it makes more sense. For net4.0, MEF is a no-brainer but we’ll cross that bridge when we get to it. …

Note that the Analyzer takes two enumerable lists – one for member-specific attribute validators and one for class-specific attribute validators. These are denoted by the interface they are derived from: ICustomValidationByTypeAnalyzer or ICustomValidationByMemberAnalyzer.

Understanding How To Write Custom Validators

Writing the custom validators is straightforward and simple. Just implement the ICustomValidator interface on your validator according to the attribute site target. From there, you declare:

1. The type of the attribute that you are testing.
2. The actual test using the contextual information about the attribution site.

And there you have it. Check out the code below to see the custom validator in detail. It’s not fully fleshed out, but provides a good boilerplate as such. Now each time you run your unit tests, your models will be validated and you won’t have to

   1: // since the matching attribute only applies at the class level, we'll use the ByTypeAnalyzer variant

   2: public class PropertiesMustMatchValidator : ICustomValidationByTypeAnalyzer 

   3: {

   4:     #region ICustomValidationByTypeAnalyzer Members

   5:  

   6:     /// <summary>

   7:     /// Here we'll test that the property exists on the object according to our criteria

   8:     /// </summary>

   9:     /// <param name="candidateType">attributed class</param>

  10:     /// <param name="propertyName">property name to look for</param>

  11:     /// <returns></returns>

  12:     private bool PublicPropertyExists(Type candidateType, string propertyName)

  13:     {

  14:         return candidateType.GetProperty(propertyName, BindingFlags.GetProperty | BindingFlags.Public | BindingFlags.Instance) != null;

  15:     }

  16:  

  17:     /// <summary>

  18:     /// Responsible for raising errors according to the tests being performed on

  19:     /// the PropertiesMustMatch validator

  20:     /// </summary>

  21:     /// <param name="candidateType">candidate class type information</param>

  22:     /// <param name="validationAttribute">the validation attribute instance that is being tested</param>

  23:     /// <param name="enforceResourceMode">whether to enforce valid resourcing</param>

  24:     /// <returns>list of error error information</returns>

  25:     public IEnumerable<CustomValidationInfo> Validate(Type candidateType, ValidationAttribute validationAttribute, bool enforceResourceMode)

  26:     {

  27:         var list = new List<CustomValidationInfo>();

  28:         var matchAttribute = validationAttribute as PropertiesMustMatchAttribute;

  29:  

  30:         if (!String.IsNullOrEmpty(matchAttribute.OriginalProperty) &&

  31:             !String.IsNullOrEmpty(matchAttribute.ConfirmProperty))

  32:         {

  33:             if( ! PublicPropertyExists(candidateType, matchAttribute.OriginalProperty))

  34:                 list.Add(new CustomValidationInfo(ValidationStatus.DeclarationError, string.Format(CultureInfo.InvariantCulture, "OriginalProperty named '{0}' does not exist on the object", matchAttribute.OriginalProperty)));

  35:  

  36:             if( ! PublicPropertyExists(candidateType, matchAttribute.ConfirmProperty))

  37:                 list.Add(new CustomValidationInfo(ValidationStatus.DeclarationError, string.Format(CultureInfo.InvariantCulture, "ConfirmProperty named '{0}' does not exist on the object", matchAttribute.ConfirmProperty)));

  38:  

  39:             // more tests could be added here - validating type, other required attribute declarations, etc

  40:         }

  41:         else

  42:             list.Add(new CustomValidationInfo(ValidationStatus.DeclarationError,

  43:                                               "Original Property and/or ConfirmPropert is not set"));

  44:  

  45:         return list;

  46:  

  47:     }

  48:  

  49:     /// <summary>

  50:     /// the attribute type that this validator is testing

  51:     /// </summary>

  52:     /// <returns>Type info on the targeted attribute</returns>

  53:     public Type TargetType()

  54:     {

  55:         return typeof (PropertiesMustMatchAttribute);

  56:     }

  57:  

  58:     #endregion

  59: }

Summary

And there you have it. Check out the code above to see the custom validator in detail. It’s not fully fleshed out, but provides a good boilerplate as such. Now each time you run your unit tests, your models will be validated and you won’t have to ‘wait until runtime’ to discover the quality of your model attributions.