Ayende @ Rahien

Refunds available at head office

The best multi threading debugging tool is Microsoft Excel

In any system that gets to a certain size, especially one that is multi threaded, there are a certain class of bugs that are really a bitch to figure out.

They are usually boil down to some sort of a race condition. I have been doing that recently, trying to fix a bunch of race conditions in RavenDB. The problem with race conditions is that they are incredibly hard to reproduce, and even when you can reproduce them, you can’t really debug them.

I came up with the following test harness to try to narrow things down. Basically, create a test case that sometimes fails, and run it a thousand times.

Odds are that you’ll get the failure, but that isn’t the important bit. The important bit is that you setup logging properly. In my case, I set things up so the logging output to CSV and each test run had a different file.

This gives me output that looks like this:

23:54:34.5952,Raven.Database.Server.HttpServer,Debug,Request #  10: GET     -    12 ms - <default>  - 200 - /indexes/dynamic/Companies?query=&start=0&pageSize=1&aggregation=None,,11
23:54:34.5952,Raven.Client.Document.SessionOperations.QueryOperation,Debug,"Stale query results on non stale query '' on index 'dynamic/Companies' in 'http://reduction:8079/', query will be retried, index etag is: bcf0fed1-d975-43b4-bfb7-65221ef06b99",,1
23:54:34.5952,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 10, for: TasksExecuter, will wait for additional work",,6
23:54:34.5952,Raven.Database.Indexing.WorkContext,Debug,Incremented work counter to 11 because: WORK BY IndexingExecuter,,12
23:54:34.5952,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 11, for: TasksExecuter, will wait for additional work",,6
23:54:34.5952,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 11, for: ReducingExecuter, will wait for additional work",,21
23:54:34.5952,Raven.Database.Indexing.WorkContext,Debug,"No work was found, workerWorkCounter: 11, for: IndexingExecuter, will wait for additional work",,12
23:54:34.6952,Raven.Client.Document.SessionOperations.QueryOperation,Debug,Executing query '' on index 'dynamic/Companies' in 'http://reduction:8079/',,1
23:54:34.8172,Raven.Database.Indexing.Index.Querying,Debug,Issuing query on index Temp/Companies for all documents,,11
23:54:34.8172,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'companies/1' was found,,11

Not really helpful in figure out what is going on, right? Except for one tiny thing, we load them to Excel, and we use conditional formatting to get things to look like this:

image

The reason this is so helpful? You can actually see the threads interleaving. This usually help me get to roughly the right place, and then I can add additional logging so I can figure out better what is actually going on.

I was able to detect and fix several race conditions using this approach.

And yes, I know that this is basically printf() debugging. But at least it is printf() debugging with pretty colors.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Frank Quednau
05/09/2012 09:27 AM by
Frank Quednau

Just wondering, have you ever given CHESS (http://research.microsoft.com/en-us/projects/chess/) a spin, considering that kind of problem could be right up its alley?

Ayende Rahien
05/09/2012 09:30 AM by
Ayende Rahien

Frank, We found multiple bugs in CHESS when we tried to run it on RavenDB

tobi
05/09/2012 09:51 AM by
tobi

CHESS is, unfortunately, abandoned. The approach this tool uses is one of the best available. So easy to use and so thorough.

Thomas Freudenberg
05/09/2012 11:13 AM by
Thomas Freudenberg

Have you ever tried log4view (http://www.log4view.com/)? It provides a nice UI to analyze log outputs produced by log4net, nlog or whatever. It has one or another memory leak, but except that it's great. Using log4view you could skip the CVSs but see the logs live using a TCP appender.

Iain
05/09/2012 01:07 PM by
Iain

As a (IMHO, better) alternative to Excel for this sort of data analysis, you could try Google Refine http://code.google.com/p/google-refine/

The facet based interface allows you to quickly burrow into the data and filter out the irrelevant.

Chris
05/09/2012 02:19 PM by
Chris

Sounds like the uncertainty principle at work!

Gokul
05/09/2012 04:33 PM by
Gokul

IMHO you can also try using logparserlizard. You may have to write a simple regex but once you get it going it's very helpful. http://www.lizard-labs.net/logparserlizard.aspx

magellings
05/09/2012 05:56 PM by
magellings

Visual Studio 2010 has a lot of support for debugging multi-threaded code. It was one of the main selling points to the 2010 IDE. Perhaps it doesn't help with race conditions, but out of curiousity did you dig into what it all provides to see if it can?

 Delay
05/10/2012 04:40 PM by
Delay

Another tool you might consider is TextAnalysisTool.NET: http://blogs.msdn.com/b/delay/archive/2007/06/21/powerful-log-file-analysis-for-everyone-releasing-textanalysistool-net.aspx

It allows you to easily filter lines in and out of view based on matching text, patterns, and markers. I wrote it a long time ago for very similar purposes. :)

Clinton
05/10/2012 08:55 PM by
Clinton

CHESS is open source: https://chesstool.codeplex.com/

Comments have been closed on this topic.