Visual Studio 2005: Unfit for Testing

time to read 5 min | 930 words

I'm not sure who is in fault here. If it is me or if it's something in VS. I didn't get any training, or read any books, before moving to VS 2005. And I am certainly struggling with it. I'd a comfortable set of tools in VS 2003, which I knew and loved. And VS 2005 has (on the surface) superior tools. But I keep running into stupid things that making it impossible for me to use them.

I've used VS testing capabilities for less than a single day, and I can tell you right now whoever wrote them left gaping holes in the implementation. Comparing VS 2005 Unit Testing (From Microsoft, supposedly well researched and well tested) vs. the open source unit testing frameworks (which doesn't have nearly the same resources as Microsoft) the open source wins hands down.

Oh, the VS 2005 UI is much slicker, and you need to combine a lot of tools to get the same results as what VS 2005 is offering, but the problem is that VS 2005 unit testing just doesn't work!

You can't use the abstract test fixture pattern! They know about it since January, but don't seem willing to change that! Another request in April got a "By Design" status and was postphoned to the next release. They also suggest to duplicate your code in order to cover this scenario.

I don't know about Microsoft, but when I've a complex class library, I want to know that a change to a base class (with potentially dozens of children, and very complex logic) is done in one place only. I believe this is called the DRY principal, which I hear has some importance in a distant developer cult located in the Himallaya. Obviously Microsoft thinks otherwise.

The microsoft solution to testing such a class is:

  • Change all the hundreds of concrete tests that you've that refer to a spesific functionality in the base class. (Since each concrete class gets its own copy of the base class tests.) - Time estimate: days.
  • Run the tests and see all the tests breaks.
  • Make the change to the base class.
  • Run the tests again and watch all the tests that you forgot to modify breaks.

The xUnit solution for testing such a class is:

  • Make a change to the single test in the abstract test class. - Time estimate: minutes.
  • Run and see it breaks.
  • Make the change to the base class.
  • Run the tests and get the green bar.

Can you spot the difference?

It's a simple change in the test runner, you need to drop the DeclaredOnly flag, but it's probably a major change with regard to the accopanying toolkit.

This comes after I discovered other major flaw in the product, the following code is a passing test:

[TestMethod(), ExpectedException(typeof(ArgumentNullException),"Blah")] 
public void MethodUnderTest() 

  throw new ArgumentNullException("1234"); 

Can you spot the huge bug? See what the exception message is? And what is the expected exception message it?

Again, this is from a very short introduction to the VS 2005 Unit Testing capabilities, and I'm really impressed with the abilities of the UI although there are some strange holes there as well. Where is the ability to point at a test/fixture and say run those tests? The functionality exists, but it's a bit involved procedure.

Nevertheless, I'm forced to conclude that VS 2005 as it stands today is unfit to be used as a serious Unit Testing framework. It has a lot of other capabilities that I didn't even began to explore (Load testing, web testing, etc). But for unit testing it simply is an unacceptable hurdle.

The rules for Test Driven Development are:

  1. Write a small test.
  2. Write just enough code to make it pass.
  3. Remove Duplication

You can't do that in VS 2005.

By the way, I'm using the release candidate, so it's pretty certain that this is the way it will go to market, which is really sad.