Fixing test failures using binary search

time to read 3 min | 404 words

With Rhino Queues, I have a very serious problem. While all the tests would run if they were run individually, running them in concrete would cause them to fail. That is a pretty common case of some test affecting the global state of the system (in fact, the problem is one test not releasing resources and all other tests failing because they tried to access a locked resource).

The problem is finding out which. By the way, note that I am making a big assumption here, that there is only one test that does so, but for the purposes of this discussion, it doesn’t matter, the technique works for both options.

At the heart of it, we have the following assumption:

If you have tests that cause failure, and you don’t know which, start by deleting tests.

In practice, it means:

  1. Run all unit tests
  2. If the unit tests failed:
    • Delete a unit test file with all its tests
    • Go to first step
  3. If the unit tests passed:
    • Revert deletion of a unit test file in reverse order to their deletion
    • Go to step 1

When you have only a single file, you have pinpointed the problem.

In my case, unit tests started to consistently pass when I deleted all of those ( I took a shortcut and delete entire directories ):


So I re-introduced UsingSubQueues, and… the tests passed.

So I re-introduced Storage/CanUseQueue and… the tests passed.

This time, I re-introduced Errors, because I have a feeling that QueueIsAsync may cause that, and… the tests FAILED.

We have found the problem. Looking at the Errors file, it was clear that it wasn’t disposing resources cleanly, and that was a very simple fix.

The problem wasn’t the actual fix, the problem was isolating it.

With the fix… the tests passed. So I re-introduced all the other tests and run the full build again.

Everything worked, and I was on my way :-)