Ayende @ Rahien

Refunds available at head office

Fixing up the build process

There is a big problem in the RavenDB build process. To be rather more exact, there is a… long problem in the RavenDB build process.

image

As you can imagine, when the build process run for that long, it doesn’t' get run too often. We already had several runs of “let us optimize the build”. But… the actual reason for the tests taking this long is a bit sneaky.

image

To save you from having to do the math, this means an average of 1.15 seconds per test.

In most tests, we actually have to create a RavenDB instance. That doesn’t take too long, but it does take some time. And we have a lot of tests that uses the network, because we need to test how RavenDB works on the wire.

From that perspective, it means that we don’t seem to have any real options. Even if we cut the average cost of running the tests by half, it would still be a 30 minutes build process.

Instead, we are going to create a layered approach. We are going to freeze all of our existing tests, move them to an Integration Tests project. We will create a small suite of tests that cover just core stuff with RavenDB, and use that. Over time, we will be adding tests to the new test project. When that becomes too slow, we will have another migration.

What about the integration tests? Well, those will be run solely by our build server, and we will setup things so we can automatically test when running from our own forks, not the main code line.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Johannes Rudolph
05/20/2013 09:48 AM by
Johannes Rudolph

Have you though about running your test suites in parallel? We're doing that quite heavily here. It's often difficulat to parallelize a single test suite, but if you have multiple of them you can easily run them in parallel.

MSBuild does a good job orchestrating that, there are a few caveats though (e.g. http://jorudolph.wordpress.com/2011/04/15/multiple-test-runner-scenarios-in-msbuild/)

tobi
05/20/2013 10:21 AM by
tobi

I once had the idea of running a randomly chosen subset of 10% of the tests. That would cover most outrageous bugs during development.

Luke
05/20/2013 10:56 AM by
Luke

How important is hitting the actual file system? Could you write to an in memory file system instead and sub in the real thing for a daily build?

Bjorn Coltof
05/20/2013 12:02 PM by
Bjorn Coltof

Mocks could help with this ;)

Matt Johnson
05/20/2013 08:59 PM by
Matt Johnson

It would be nice if you could specify in each test definition what kind of parallelism it could handle. Tests that just need an in-memory database with no http could probably run several at a time in parallel. Other tests, like those testing replication over specific http ports, would run serially. It would be better than forcing them all to run sequentially.

Ken Egozi
05/21/2013 06:37 AM by
Ken Egozi

Making a build faster turns kittens into unicorns!

two questions here: 1. why the separate test project? can't you parameterize the test run, to only include the long-running tests when env=build-server or something of sorts?

  1. The split to Integration ans Unit is too by-the-book, yet not as practical as it could be - some integration tests might be fast, some unit tests might be slow, and some (many) tests are difficult to categorize as one or the other without getting into unneeded religiousness. instead - just call them "slow" and "quick" tests. bam.
Ken Egozi
05/21/2013 06:39 AM by
Ken Egozi

lololo

two questions - 1 and 1.

by the time you're 35, and after having put two kids to bed and then reviewed and wrote code for a couple of hours, 11:30pm is late ..

Ayende Rahien
05/21/2013 08:49 AM by
Ayende Rahien

Johannes, We have a lot of tests that require stuff that cannot be run concurrently. For example, all our tests that test the REST API require a port, etc.

Ayende Rahien
05/21/2013 08:50 AM by
Ayende Rahien

Luke, We are a database, touching the FS is something that is quite important for us. An expensive test we have, for example, is what happens when we corrupt the files on the FS, how do we recover from that?

Patrick Huizinga
05/21/2013 09:50 AM by
Patrick Huizinga

Ayende,

It seems Luke asked if you could use an in memory file system for such tests. You don't need a piece of metal to test the handling of corrupted files, right?

Dylan Smith
05/22/2013 02:42 AM by
Dylan Smith

What I commonly do is have 2 "CI" builds. One that kicks off on check-in and runs in some short time span (say 5 mins or less). This is responsible for performing the most important quality checks against the changes (what is most important will be different from team to team, but it usually includes some sub-set of fast and/or important tests).

If/when that build succeeds another build gets kicked off immediately afterwards that runs all quality checks/tests, and this may take upwards of an hour.

This way the developer gets the most important feedback quickly, and then even more feedback shortly thereafter.

In complex projects if the 2nd build gets to be much more than an hour I may introduce a 3rd tier build that runs as a nightly build.

Comments have been closed on this topic.