Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,131 | Comments: 45,568

filter by tags archive

Fixing up the build process

time to read 2 min | 325 words

There is a big problem in the RavenDB build process. To be rather more exact, there is a… long problem in the RavenDB build process.


As you can imagine, when the build process run for that long, it doesn’t' get run too often. We already had several runs of “let us optimize the build”. But… the actual reason for the tests taking this long is a bit sneaky.


To save you from having to do the math, this means an average of 1.15 seconds per test.

In most tests, we actually have to create a RavenDB instance. That doesn’t take too long, but it does take some time. And we have a lot of tests that uses the network, because we need to test how RavenDB works on the wire.

From that perspective, it means that we don’t seem to have any real options. Even if we cut the average cost of running the tests by half, it would still be a 30 minutes build process.

Instead, we are going to create a layered approach. We are going to freeze all of our existing tests, move them to an Integration Tests project. We will create a small suite of tests that cover just core stuff with RavenDB, and use that. Over time, we will be adding tests to the new test project. When that becomes too slow, we will have another migration.

What about the integration tests? Well, those will be run solely by our build server, and we will setup things so we can automatically test when running from our own forks, not the main code line.


Johannes Rudolph

Have you though about running your test suites in parallel? We're doing that quite heavily here. It's often difficulat to parallelize a single test suite, but if you have multiple of them you can easily run them in parallel.

MSBuild does a good job orchestrating that, there are a few caveats though (e.g. http://jorudolph.wordpress.com/2011/04/15/multiple-test-runner-scenarios-in-msbuild/)


I once had the idea of running a randomly chosen subset of 10% of the tests. That would cover most outrageous bugs during development.


How important is hitting the actual file system? Could you write to an in memory file system instead and sub in the real thing for a daily build?

Bjorn Coltof

Mocks could help with this ;)

Matt Johnson

It would be nice if you could specify in each test definition what kind of parallelism it could handle. Tests that just need an in-memory database with no http could probably run several at a time in parallel. Other tests, like those testing replication over specific http ports, would run serially. It would be better than forcing them all to run sequentially.

Ken Egozi

Making a build faster turns kittens into unicorns!

two questions here: 1. why the separate test project? can't you parameterize the test run, to only include the long-running tests when env=build-server or something of sorts?

  1. The split to Integration ans Unit is too by-the-book, yet not as practical as it could be - some integration tests might be fast, some unit tests might be slow, and some (many) tests are difficult to categorize as one or the other without getting into unneeded religiousness. instead - just call them "slow" and "quick" tests. bam.
Ken Egozi


two questions - 1 and 1.

by the time you're 35, and after having put two kids to bed and then reviewed and wrote code for a couple of hours, 11:30pm is late ..

Ayende Rahien

Johannes, We have a lot of tests that require stuff that cannot be run concurrently. For example, all our tests that test the REST API require a port, etc.

Ayende Rahien

Luke, We are a database, touching the FS is something that is quite important for us. An expensive test we have, for example, is what happens when we corrupt the files on the FS, how do we recover from that?

Patrick Huizinga


It seems Luke asked if you could use an in memory file system for such tests. You don't need a piece of metal to test the handling of corrupted files, right?

Dylan Smith

What I commonly do is have 2 "CI" builds. One that kicks off on check-in and runs in some short time span (say 5 mins or less). This is responsible for performing the most important quality checks against the changes (what is most important will be different from team to team, but it usually includes some sub-set of fast and/or important tests).

If/when that build succeeds another build gets kicked off immediately afterwards that runs all quality checks/tests, and this may take upwards of an hour.

This way the developer gets the most important feedback quickly, and then even more feedback shortly thereafter.

In complex projects if the 2nd build gets to be much more than an hour I may introduce a 3rd tier build that runs as a nightly build.

Comment preview

Comments have been closed on this topic.


  1. RavenDB Conference 2016–Slides - 5 hours from now
  2. Proposed solution to the low level interview question - about one day from now

There are posts all the way to Jun 02, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats