http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660I was hoping you would say that. I have always used Rhino in the past but on this client site they are using Unity+Moq. Hmmmm.... It may be time to change back before we have too many unit tests written, or just use Rhino for my multi-threaded ones. Thanks http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment12http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment12Thu, 16 Sep 2010 11:10:39 GMTRob, Use a mocking library that is thread safe (Rhino Mocks)? http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment11http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment11Thu, 16 Sep 2010 10:45:34 GMT@Ayende, "Usually, this is an indication of a problem with the code that is exposed through multi threading." What do you do when the exception is in the mock library and not your code? I am passing a Moq service mock into multiple threads. It intermittently blows up because it seems that that Setup (Expect) delegates use a non-synchronized List (you get Index exception on the List). Now, I know my object under test works because I never get an exception from there. What should I do, stop doing the mutli-threaded tests (single threaded tests all work fine), create a custom thread-safe mock for myself, or...? http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment10http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment10Thu, 16 Sep 2010 08:54:05 GMT[Repeat(10000)] and [ThreadedRepeat(100)] are generally very effective. http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment9http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment9Wed, 15 Sep 2010 18:13:19 GMTI once faced a problem where an unknown test failed once in a blue moon with a -100 error (basically: Exception occurred on secondary thread, causing NUnit to exit abnormally without any information). The problem was that the tests were soooooooooo slow. I think when I rolled into town, the 10k 'unit' tests would take well over 10 minutes to run, but the -100 popped up once in a blue moon. As a result, it'd take literally days to make the failure occur and, when you did, there'd be no chance of knowing which assembly, never mind test, was causing it. I was so frustrated by it that I ended up writing a script to perform a binary search. It'd omit half of the test assemblies, run the other half in a loop for hours, analyse the exit codes and dump out the assemblies present when the failure occured. It'd then take the subset of failing assemblies and repeat the process. I set it running on a spare machine over the weekend and came back to find the assembly had been identified. After that it was just a case of running half of the tests in the assembly in a loop and repeating the process. It took days, but we eventually found the culprit. This is why paying attention to broken windows is a 'good thing' for your mental health... I just about killed someone :) http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment8http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment8Wed, 15 Sep 2010 18:05:58 GMTI believe that suspending an resuming threads at random (done by a dedicated controller thread) can help diagnose those issues. It is a piece of write-once infrastructure. http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment7http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment7Wed, 15 Sep 2010 17:05:20 GMTAlbert Einstein once said "The definition of insanity is doing the same thing over and over again and expecting different results". Who's insane? http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment6http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment6Wed, 15 Sep 2010 13:57:41 GMTLove it! I debug this way; I'm not a debugger expert, so I use Console.WriteLine() frequently, and trace through the code in my head. I have found that "occasionally failing" tests are almost always related to threading code. I have learned to use an abstraction for launching threads and utilize a synchronized version of the thread starter for unit testing. http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment5http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment5Wed, 15 Sep 2010 13:56:34 GMTGoogle Microsoft Chess, haven't had time to check it out myself yet but could help with these issues. I pasted the url in a previous comment but it got flagged as spam. The problem is that event with a loop you might not hit the issue, there are so many variables to consider like cpu architecture, system load etc when it comes to mutli threading bugs. http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment4http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment4Wed, 15 Sep 2010 12:46:54 GMT'Usually, this is an indication of a problem with the code that is exposed through multi threading' Another reason can be using DateTime.Now and its ticks modulo sth in a legacy code;-) http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment3http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment3Wed, 15 Sep 2010 11:50:51 GMTIndeed, using random numbers in unit tests is bad; use arbitrary numbers instead. @Ayende, you can probably debug the tests by using your Main() function and setting the debugger to stop on exceptions; or would even having a debugger connected stop the test from working in this case? http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment2http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment2Wed, 15 Sep 2010 10:44:17 GMTI can offer a funny story about occasionally failing tests. One of the tests in my team a couple of years ago was failing like once a day. Really random. It either did on a developer workstation or the build server. One day I got pissed at it and opened the code... I discovered an elaborate scheme to select a random table cell to use for testing. Of course it had a bug that manifested only when a specific cell was chosen by the randomizer. Our very own Russian roulette :-) Be afraid of people that use random numbers in unit tests. Be very afraid! http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment1http://ayende.com/4630/how-to-reproduce-an-occasionally-failing-test#comment1Wed, 15 Sep 2010 10:30:17 GMT