Ayende @ Rahien

Refunds available at head office

Bisecting RavenDB

One of the results of expanding the RavenDB’s team is that we have to deal with a lot more pull requests. We have individual developers working on their features, and submitting pull requests on a daily basis. That usually means that I have to merge anything between ten to thirty pull requests per day.

Given our test suite runtime, it isn’t really reasonable to run the full test suite after each pull request, so we merge them all, review them, and then run the tests. The problem then is that if there was a test failure… which pull request, and which commit cause it?

That is why git has the wonderful git bisect command. And we have taken advantage of that to enable the following development time feature:

./bisect.ps1 HEAD HEAD~10 General_Initialized_WithoutErrors

It will output something like:

631caa60719b1a65891eec37c4ccb4673eb28d02 is the first bad commit
commit 631caa60719b1a65891eec37c4ccb4673eb28d02
Author: Pawel Pekrol
Date: Fri May 9 08:57:55 2014 +0200
another temp commit

Pin pointing exactly where the issue is.

And yes, this is here as a post primarily because I might forget about it.

Comments

Charles Bryant
05/21/2014 12:56 PM by
Charles Bryant

I know this post is for your memory purposes, but this is very interesting to me. Can you reveal a little of the secret magic in that PowerShell script? What is your general algorithm to find the culprit? What did you do to identify that commit?

Thanks

dorony
05/21/2014 03:45 PM by
dorony

May I ask why isn't everyone working on the same branch? Why deal with constant merges?

Jeremy Wiebe
05/21/2014 04:01 PM by
Jeremy Wiebe

@Charles Bryant: git bisect supports a script mode where you can pass it a script that returns 0 for success and non-zero for failure. Git bisect will then do the bisect work, running the script at each commit it arrives at and deciding if the commit is good or bad based on the result of running the script. I would guess Ayende has hooked up the automated tests to the script so that if a test fails it returns non-zero (thus telling git bisect that the commit is broken).

There's a good discussion of 'git bisect' here: http://git-scm.com/book/en/Git-Tools-Debugging-with-Git (there's a small blurb about the scripting support at the end of this page).

Ayende Rahien
05/21/2014 04:31 PM by
Ayende Rahien

Charles, We are just shelling out to git bisect, with a script of our own that knows how to build and run our project.

Ayende Rahien
05/21/2014 04:32 PM by
Ayende Rahien

Dorony, We have up to 10 people working on RavenDB concurrently. Even with everyone starting from the same branch in the morning, by evening, they all made their own changes.

Damian Hickey
05/22/2014 05:06 PM by
Damian Hickey

Given our test suite runtime, it isn’t really reasonable to run the full test suite after each pull request

Erm.. Teamcity? http://blog.jetbrains.com/teamcity/2013/02/automatically-building-pull-requests-from-github-with-teamcity/ . This is also great to give a contributor feedback that their PR didn't break anything before you look at it.

If the build queue grows, just get more agent licences and use EC2 integration for automagic scaling. http://www.jetbrains.com/teamcity/features/amazon_ec2.html

(I've been harping on to you guys about a proper CI setup for a while now, I won't mention it again).

Ayende Rahien
05/23/2014 06:52 AM by
Ayende Rahien

Damian, We have roughly 10 PR per day, often more than that. Our full build process take too long to run this 10 times a day, and having 10 build servers is a bit much.

Damian Hickey
05/23/2014 08:47 AM by
Damian Hickey

Oren, I'm not buying that. It's a highly parallel problem - it's just a single AMI that teamcity will automatically spin up/down instances of. Given the number of people and PRs 10 build agents running in parallel is really not much at all. That'll be the C in CI ;) (Also the TC->Github PR notification integration is useful.)