http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660zvolkov, It means disable a big parts of the tests, run the test. Enable some tests, run the test http://ayende.com/4014/fixing-test-failures-using-binary-search#comment15http://ayende.com/4014/fixing-test-failures-using-binary-search#comment15Mon, 18 May 2009 20:48:38 GMTI thought you said you used binary search? I mean it's pretty obvious but some people may find the title confusing... The logic above is linear: you just remove a test after test. Instead your logic should be "recursively disable half of the tests and see if that helped", right? http://ayende.com/4014/fixing-test-failures-using-binary-search#comment14http://ayende.com/4014/fixing-test-failures-using-binary-search#comment14Mon, 18 May 2009 20:46:04 GMTActually, it is a horrible idea. The problem is that when you DO have such a problem, you have a really hard time figuring it out. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment13http://ayende.com/4014/fixing-test-failures-using-binary-search#comment13Sat, 16 May 2009 12:53:33 GMTSounds like it might be a good feature for test frameworks to not run their tests in a pre-determined order? http://ayende.com/4014/fixing-test-failures-using-binary-search#comment12http://ayende.com/4014/fixing-test-failures-using-binary-search#comment12Sat, 16 May 2009 12:45:35 GMTI'm not surprised that running your tests in concrete would cause them to fail :) http://ayende.com/4014/fixing-test-failures-using-binary-search#comment11http://ayende.com/4014/fixing-test-failures-using-binary-search#comment11Sat, 16 May 2009 05:58:45 GMTGerardo, Order, that was the problem. That test went last on the queue, for some reason. And other tests went after it didn't use the same thing. It was only when I created some tests that did use it that it started failing, and that was long after that. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment10http://ayende.com/4014/fixing-test-failures-using-binary-search#comment10Sat, 16 May 2009 04:28:53 GMTJoe, Those are integration tests. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment9http://ayende.com/4014/fixing-test-failures-using-binary-search#comment9Sat, 16 May 2009 04:16:10 GMTI wonder why didn't you spot the problem when you wrote (and ran) the test that didn't release the file. It seems, from your description, that that particular test was written some time ago. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment8http://ayende.com/4014/fixing-test-failures-using-binary-search#comment8Sat, 16 May 2009 01:38:14 GMTThe last time I had a problem like this, I made a change to the test runner I was using (in this particular case, python's unittest module) to let me binary search my full set of tests. It ended up being worthwhile since this sort of problem can come up often, and having to do the search manually is pretty painful. I wouldn't be surprised if it was pretty easy to do the same in NUnit (or whatever runner you happen to be using). http://ayende.com/4014/fixing-test-failures-using-binary-search#comment7http://ayende.com/4014/fixing-test-failures-using-binary-search#comment7Sat, 16 May 2009 00:30:13 GMT@Ayende I think you may have missed an abstraction. Have you thought of getting rid of the specific dependency (an interface) on the file for these tests and application? From the follow on comments you stated that the application maintains the file during it's lifetime. Do you not want your tests to reflect their real-world usage? Dependencies of resources and primitives in tests, to me have a code-smell of state based testing. I believe that the failing tests were indicating changes to the code are required. Fixing the tests IMHO seems a pretty anemic solution. It may come back and bite you in the ass. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment6http://ayende.com/4014/fixing-test-failures-using-binary-search#comment6Fri, 15 May 2009 19:34:15 GMTAyende, I have some similar tests (not surprisingly on a similar problem, a transaction file to be specific). I ended up supporting temp files to avoid this so there would not be dependencies between my tests. Cheers, Greg http://ayende.com/4014/fixing-test-failures-using-binary-search#comment5http://ayende.com/4014/fixing-test-failures-using-binary-search#comment5Fri, 15 May 2009 15:59:47 GMTBilly, Duh! The problem was a bug in the tests http://ayende.com/4014/fixing-test-failures-using-binary-search#comment4http://ayende.com/4014/fixing-test-failures-using-binary-search#comment4Fri, 15 May 2009 13:41:21 GMTBruno, Tests do things that the real app never does. In that example, the problem was that one of the tests wasn't releasing a file. That never happens in the app, because that file it used throughout the lifetime of the app http://ayende.com/4014/fixing-test-failures-using-binary-search#comment3http://ayende.com/4014/fixing-test-failures-using-binary-search#comment3Fri, 15 May 2009 13:40:19 GMTThis is a code smell anyway. Any tests that effect the actual outcome of other test results should be looked at and made more isolated! http://ayende.com/4014/fixing-test-failures-using-binary-search#comment2http://ayende.com/4014/fixing-test-failures-using-binary-search#comment2Fri, 15 May 2009 12:35:33 GMTYou could make your code less dependent on global state, replacing a static class with a normal class and a single static variable. Normal code uses the static but tests use the normal class, which is created for each test. In my code I usually have conflicting tcp ports. You can let the OS chose the port for a listening socket with Socket.Bind. Lastly, I think that if you have hard to pinpoint failures in your tests, you have tested and failed in the debuggeability of your system. It's time to fire up serious debuggers like WinDbg and make changes in your program so as to make determining the problem easier. If it's hard to debug unit tests is surely hard to debug the same program in production, so it pays off to have better debug procedures in place. http://ayende.com/4014/fixing-test-failures-using-binary-search#comment1http://ayende.com/4014/fixing-test-failures-using-binary-search#comment1Fri, 15 May 2009 12:31:39 GMT