http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Thanks for fixing this one Oren. Sometimes it's really hard to give you a reproduction demo without giving you the whole product. With sufficiently complex systems we don't know which part of complexity breaks raven. We just know that the issue is in Raven. Trust me when it's not that we don't try. I think first time we hit this we spent a week trying to replicate the issue in a demo app and we couldn't. That's how we figured it could've been replication (we usually don't run it locally). It does help to build local raven and debug that to see where you can trigger an issue but sometimes that's just not possible. There are still interesting threading issues that we see (we have some queries that if we run 24 threads or less with them, raven is cool, if we run 25 threads and up first Transaction breaks). These aren't as important right now but there will be more threading holes and hopefully it will be easy to reproduce in a test.http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment5http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment5Tue, 19 Feb 2013 22:42:06 GMT Ayende say's "We protected the root properties of the metadata from concurrent modifications, and most of the time, it works just fine. But the problem was that now we had a bundle that was modifying a nested array, which wasn’t protected." ...wowhttp://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment4http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment4Mon, 18 Feb 2013 15:54:50 GMTBasically, you enable it and run the test. You put a breakpoint on the point where the test failed, then (assuming it still fails), you can go backwards in time. You basically get the same debug environment as usual (with step over, step into, watch, etc.), but you can also step back in execution and redo the test. And it does the exact same thing, i.e. timing issues still happen because you're just moving the time line back and forward without actually executing anything. It's got a bit of a learning curve, but it's extremely useful, especially in these sort of circumstances. I suggest when using it to always have the Parallel Stacks window open on a second monitor at all times; it makes things somewhat easier to follow.http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment3http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment3Mon, 18 Feb 2013 15:47:03 GMTDor, I never really got IntelliTrace, and anything that has significant perf cost is going to be problematic for multi threaded issues.http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment2http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment2Mon, 18 Feb 2013 15:09:40 GMTTry using IntelliTrace (part of Visual Studio 2012 Ultimate) next time. It can record events (you can tell it which events to record), or even every single function call (which is useful for tests, but horribly slow for any interactive software), and then you can debug multi-threaded code much more easily.http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment1http://ayende.com/160903/bug-hunting-in-a-massively-multi-threaded-environment#comment1Mon, 18 Feb 2013 14:15:34 GMT