Unit Test filtering for TFS builds using Test Explorer in VS 2012

One of the major new features in Visual Studio 2012 is the Text Explorer tool window, which consolidates 2010’s Test View and Test Results windows and adds support for using different testing frameworks together in one place. There are definitely some positive aspects to Test Explorer in comparison to its predecessors, but as a completely new piece of functionality it unfortunately left out some key features that were previously available.

One of the places it fell short was in filtering of tests to enable running a specific subset of the tests in your solution, especially when working with a set of tests set up for TFS’s Team Build. When working with small sets of tests it could be a minor annoyance, but working with hundreds or thousands of tests made it basically unusable. Thanks to Visual Studio’s new release model of frequent updates, these shortcomings are already starting to be addressed with November’s Update 1.

The preferred method of specifying tests to run with builds in TFS is by using attributes on test methods, specifically the Priority and TestCategory attributes. VS2010’s Test View displays a configurable grid listing all available tests in the open solution.
image

It also provides grouping using a dropdown which includes Priority among many other options.
image

A separate keyword filtering option allows filtering by many of the available column types, including Test Categories. Using these options allows easy narrowing of the list down to the same subset that is configured to run with a build.
image

In the new Test Explorer, the initial list of tests is the same as that shown in the old Test View, but most of the complex display options were missing in the RTM version.
image
Some of the capability has started to return with Update 1 in the form of awareness of Traits – a general term for descriptive test attributes. Traits can be used both for grouping and filtering but the behavior is significantly different than with Test View. Rather than grouping by specific attributes, Test Explorer can group by all Traits. This results in a strange view that shows groups for every Trait which is applied to any test, and in doing so lists duplicates of every test having more than one attribute applied.
image
So if you have a Priority and TestCategory applied to each of your tests, they will all show up twice in the grouped list, more if they have multiple TestCategory or other attributes applied. Thankfully, the runner is smart enough to only actually run each test once, but it’s not exactly intuitive as an organizational view.

Filtering behavior has similar oddities. Rather than the dropdown of available columns to filter on, the search box takes search parameters in the form of key-value pairs, which for our purposes looks like Trait:”value”. Adding more terms ANDs them all together to produce results.
image
Also annoying is that the weird grouped view still persists after filtering, so, for example, searching for a specific Priority value will still show all of the other Trait groups. It does at least provide the ability to get the list down to, for example, only those tests with Priority of 2, which was not an option in RTM.
image

Unfortunately, the generic filtering mechanism is restricted in the scope of filtering compared to Test View in some situations. When trying to filter by a TestCategory, for instance, using a Trait:”Data” search will include not only TestCategory, but also other test attributes with string arguments. So if I were to add an Owner of “Database Team” to a test, that would now get included in the results of my search for the “Data” TestCategory. This situation should be relatively easy to avoid by managing your test organization but it is something to be aware of.
image

Another caveat is the logical behavior when using multiple Trait filters. The filters are combined as a logical AND but are not differentiated as far as what they are filtering against so a filter of Trait:”Priority” Trait:”1″ means “tests with a Priority attribute AND an attribute containing a value of 1″ and not “tests with a Priority attribute with a value of 1″. The difference is subtle but can give different results. Using the Owner example again, if I use “Team 1″ and then use the above filter I now get everything with Priority of 1 (which is expected), but also anything owned by Team 1, no matter what the Priority is. Again, should be easy to avoid, but be aware of it.
image

Hopefully Test Explorer will continue to improve with each Update release. It seems like a good starting point for a reboot of VS’s unit testing interface, just not quite ready to replace everything that was removed in its favor.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>