+972 52-548-6969
The cadence of tests speed
RavenDB has quite a bit of tests. Over five thousands of them, on the last count. They test common things (like saving a document works) and esoteric things (like spatial query behavior near the equator). They are important, but they also take quite a lot of time to run. In our current setup, running the full test suite can take 3 – 6 hours, depending on which build agent is running and what is the configuration we have.
This means that in terms of actually running the tests, developers can’t really get good feedback. We have taken some steps to address that, by creating a core set of tests that will examine the major and most important features, fast, which developers can run in reasonable time as part of their work, and then we let the build server sort out the pull requests on its own time.
For RavenDB 4.0, we decided to go in a different direction. We split the tests into two separate projects. FastTests and SlowTests. The requirement for the FastTest is that they will run in under 1 minute. In order to do that we use xunit’s parallel test support, which allows us to run concurrent tests (this actually required a lot of changes in the test code, which assumed that we are running in a serial fashion and that each test is the sole owner of the entire system).
Part of making sure that we are running in under 1 minute is knowing when we breached that. Because the tests run concurrently, it is a bit difficult to know what is taking so long. Thankfully, it is easy to get a report out. We use the following script to do that.
We now have the top slowest tests (some of them takes tens of seconds to run, but we don’t usually care, as long as other test run concurrently parallel and the whole suite is fast), and we can figure out whatever we can optimize the test or move it to the SlowTest project.
Another change we made was to move a lot of the test code to use the async API, this allows us to run more tests concurrently without putting too much of a burden on the system. Our current tests times range between 30 – 55 seconds, which is enough to run them pretty much on every drop of a hat.
Comments
Thoughts on speeding up test runs with the help of the cloud? Maybe spin up 80 CPU cores instead of 8.
This could help the build server as well.
From the media coverage I gather that clouds nowadays have very low overhead ways of running a piece of code (AWS lambda and Azure equivalent). Surely other ways to do it.
Tobi, At some point, you get to diminishing returns. And you want rapid feedback, 1 minute is fine (you let it run when you are preparing the PR, and fix if there is something there).
:-)
What about the test of time?
E.g running for 24 hours to see problems with day- month- and yearshift and memory leaks. You cannot do that in one minutes unless you change datetime :-)
You made need to run for some time in order to meet replication or daily backup of database or antivirus-scan or other strange programs :-)
If you want to test with 100 users on one database you cannot go to the cloud with 100 x (one database with one user)
Carsten, Oh, we do that. We just run the tests in a loop via a PowerShell script. That gives us a better degree of confidence in the tests because there are so many runs over such long period of time, most races will be exposed.
That said, those are functional tests, they come to test that we are doing okay here. They aren't longevity / performance tests
Comment preview