Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,026 | Comments: 44,842

filter by tags archive

Bisecting RavenDB

time to read 1 min | 179 words

One of the results of expanding the RavenDB’s team is that we have to deal with a lot more pull requests. We have individual developers working on their features, and submitting pull requests on a daily basis. That usually means that I have to merge anything between ten to thirty pull requests per day.

Given our test suite runtime, it isn’t really reasonable to run the full test suite after each pull request, so we merge them all, review them, and then run the tests. The problem then is that if there was a test failure… which pull request, and which commit cause it?

That is why git has the wonderful git bisect command. And we have taken advantage of that to enable the following development time feature:

./bisect.ps1 HEAD HEAD~10 General_Initialized_WithoutErrors

It will output something like:

631caa60719b1a65891eec37c4ccb4673eb28d02 is the first bad commit
commit 631caa60719b1a65891eec37c4ccb4673eb28d02
Author: Pawel Pekrol
Date: Fri May 9 08:57:55 2014 +0200
another temp commit

Pin pointing exactly where the issue is.

And yes, this is here as a post primarily because I might forget about it.


Charles Bryant

I know this post is for your memory purposes, but this is very interesting to me. Can you reveal a little of the secret magic in that PowerShell script? What is your general algorithm to find the culprit? What did you do to identify that commit?



May I ask why isn't everyone working on the same branch? Why deal with constant merges?

Jeremy Wiebe

@Charles Bryant: git bisect supports a script mode where you can pass it a script that returns 0 for success and non-zero for failure. Git bisect will then do the bisect work, running the script at each commit it arrives at and deciding if the commit is good or bad based on the result of running the script. I would guess Ayende has hooked up the automated tests to the script so that if a test fails it returns non-zero (thus telling git bisect that the commit is broken).

There's a good discussion of 'git bisect' here: http://git-scm.com/book/en/Git-Tools-Debugging-with-Git (there's a small blurb about the scripting support at the end of this page).

Ayende Rahien

Charles, We are just shelling out to git bisect, with a script of our own that knows how to build and run our project.

Ayende Rahien

Dorony, We have up to 10 people working on RavenDB concurrently. Even with everyone starting from the same branch in the morning, by evening, they all made their own changes.

Damian Hickey

Given our test suite runtime, it isn’t really reasonable to run the full test suite after each pull request

Erm.. Teamcity? http://blog.jetbrains.com/teamcity/2013/02/automatically-building-pull-requests-from-github-with-teamcity/ . This is also great to give a contributor feedback that their PR didn't break anything before you look at it.

If the build queue grows, just get more agent licences and use EC2 integration for automagic scaling. http://www.jetbrains.com/teamcity/features/amazon_ec2.html

(I've been harping on to you guys about a proper CI setup for a while now, I won't mention it again).

Ayende Rahien

Damian, We have roughly 10 PR per day, often more than that. Our full build process take too long to run this 10 times a day, and having 10 build servers is a bit much.

Damian Hickey

Oren, I'm not buying that. It's a highly parallel problem - it's just a single AMI that teamcity will automatically spin up/down instances of. Given the number of people and PRs 10 build agents running in parallel is really not much at all. That'll be the C in CI ;) (Also the TC->Github PR notification integration is useful.)

Comment preview

Comments have been closed on this topic.


No future posts left, oh my!


  1. Technical observations from my wife (3):
    13 Nov 2015 - Production issues
  2. Production postmortem (13):
    13 Nov 2015 - The case of the “it is slow on that machine (only)”
  3. Speaking (5):
    09 Nov 2015 - Community talk in Kiev, Ukraine–What does it take to be a good developer
  4. Find the bug (5):
    11 Sep 2015 - The concurrent memory buster
  5. Buffer allocation strategies (3):
    09 Sep 2015 - Bad usage patterns
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats