+972 52-548-6969
Bisecting RavenDB
One of the results of expanding the RavenDB’s team is that we have to deal with a lot more pull requests. We have individual developers working on their features, and submitting pull requests on a daily basis. That usually means that I have to merge anything between ten to thirty pull requests per day.
Given our test suite runtime, it isn’t really reasonable to run the full test suite after each pull request, so we merge them all, review them, and then run the tests. The problem then is that if there was a test failure… which pull request, and which commit cause it?
That is why git has the wonderful git bisect command. And we have taken advantage of that to enable the following development time feature:
./bisect.ps1 HEAD HEAD~10 General_Initialized_WithoutErrors
It will output something like:
631caa60719b1a65891eec37c4ccb4673eb28d02 is the first bad commit
commit 631caa60719b1a65891eec37c4ccb4673eb28d02
Author: Pawel Pekrol
Date: Fri May 9 08:57:55 2014 +0200
another temp commit
Pin pointing exactly where the issue is.
And yes, this is here as a post primarily because I might forget about it.
Comments
I know this post is for your memory purposes, but this is very interesting to me. Can you reveal a little of the secret magic in that PowerShell script? What is your general algorithm to find the culprit? What did you do to identify that commit?
Thanks
May I ask why isn't everyone working on the same branch? Why deal with constant merges?
@Charles Bryant: git bisect supports a script mode where you can pass it a script that returns 0 for success and non-zero for failure. Git bisect will then do the bisect work, running the script at each commit it arrives at and deciding if the commit is good or bad based on the result of running the script. I would guess Ayende has hooked up the automated tests to the script so that if a test fails it returns non-zero (thus telling git bisect that the commit is broken).
There's a good discussion of 'git bisect' here: http://git-scm.com/book/en/Git-Tools-Debugging-with-Git (there's a small blurb about the scripting support at the end of this page).
Charles, We are just shelling out to git bisect, with a script of our own that knows how to build and run our project.
Dorony, We have up to 10 people working on RavenDB concurrently. Even with everyone starting from the same branch in the morning, by evening, they all made their own changes.
Pawel called out. Ouch. ;-]
Erm.. Teamcity? http://blog.jetbrains.com/teamcity/2013/02/automatically-building-pull-requests-from-github-with-teamcity/ . This is also great to give a contributor feedback that their PR didn't break anything before you look at it.
If the build queue grows, just get more agent licences and use EC2 integration for automagic scaling. http://www.jetbrains.com/teamcity/features/amazon_ec2.html
(I've been harping on to you guys about a proper CI setup for a while now, I won't mention it again).
Damian, We have roughly 10 PR per day, often more than that. Our full build process take too long to run this 10 times a day, and having 10 build servers is a bit much.
Oren, I'm not buying that. It's a highly parallel problem - it's just a single AMI that teamcity will automatically spin up/down instances of. Given the number of people and PRs 10 build agents running in parallel is really not much at all. That'll be the C in CI ;) (Also the TC->Github PR notification integration is useful.)
Comment preview