Repeatable random tests
Testing our software is something that we take very serious. And in some cases, we want to go beyond testing stuff that we know. We want to test random stuff. For example, if we add 10,000 documents, then remove every 17th, what happens? Is there any differences in behavior, performance, etc?
It is easy to do random stuff, of course. But that leads to an interesting case. As long as the tests are passing, you can pat yourself on the back: “We have done good, and everything works as it should”.
But when something fails… well, the only thing that you know is that something did fail. You don’t have a way to reproduce this. Because the test is… random.
In order to handle that, we wrote the following code:
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)] public class InlineDataWithRandomSeed : DataAttribute { public InlineDataWithRandomSeed(params object[] dataValues) { this.DataValues = dataValues ?? new object[] {null}; } public object[] DataValues { get; set; } public override IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes) { var objects = new object[DataValues.Length+1]; Array.Copy(DataValues,0,objects,0, DataValues.Length); objects[DataValues.Length] = Environment.TickCount; yield return objects; } }
This is using XUnit, which gives us the ability to add a seed to the test. Let us see how the test looks:
And this is what this looks like when it runs:
When we have a failure, we know what the seed is, and we can run the test with that seed, and see what exactly happened there.
Comments
Have you considered using something like FsCheck? I don't have much experience in it. Just read some stuff about it and property-based testing and wonder if it's applicable here.
http://fsharpforfunandprofit.com/posts/property-based-testing/
Reminds me a bit of AutoFixture and its ability to auto-generate random but consistent test data. I haven't had the chance to use it in anger yet but it's an interesting idea. https://github.com/AutoFixture/AutoFixture/wiki/Cheat-Sheet
Comment preview