Structuring your Unit Tests, why?
I am a strong believer in automated unit tests. And I read this post by Phil Haack with part amusement and part wonder.
RavenDB currently has close to 1,400 tests in it. We routinely ask for failing tests from users and fix bugs by writing tests to verify fixes.
But structuring them in terms of source code? That seems to be very strange.
You can take a look at the source code layout of some of our tests here: https://github.com/ayende/ravendb/tree/master/Raven.Tests/Bugs
It is a dumping ground, basically, for tests. That is, for the most part, I view tests as very important in telling me “does this !@#! works or not?” and that is about it. Spending a lot of time organizing them seems to be something of little value from my perspective.
If I need to find a particular test, I have R# code browsing to help me, and if I need to find who is testing a piece of code, I can use Find References to get it.
At the end, it boils down to the fact that I don’t consider tests to be, by themselves, a value to the product. Their only value is their binary ability to tell me whatever the product is okay or not. Spending a lot of extra time on the tests distract from creating real value, shippable software.
What I do deeply care about with regards to structuring the tests is the actual structure of the test. It is important to make sure that all the tests looks very much the same, because I should be able to look at any of them and figure out what is going on rapidly.
I am not going to use the RavenDB example, because that is system software and usually different from most business apps (although we use a similar approach there). Instead, here are a few tests from our new ordering system:
[Fact] public void Will_send_email_after_trial_extension() { Consume<ExtendTrialLicenseConsumer, ExtendTrialLicense>(new ExtendTrialLicense { ProductId = "products/nhprof", Days = 30, Email = "you@there.gov", }); var email = ConsumeSentMessage<NewTrial>(); Assert.Equal("you@there.gov", email.Email); } [Fact] public void Trial_request_from_same_customer_sends_email() { Consume<NewTrialConsumer, NewTrial>(new NewTrial { ProductId = "products/nhprof", Email = "who@is.there", Company = "h", FullName = "a", TrackingId = Guid.NewGuid() }); Trial firstTrial; using (var session = documentStore.OpenSession()) { firstTrial = session.Load<Trial>(1); } Assert.NotNull(ConsumeSentMessage<SendEmail>()); Consume<NewTrialConsumer, NewTrial>(new NewTrial { TrackingId = firstTrial.TrackingId, Email = firstTrial.Email, Profile = firstTrial.ProductId.Substring("products/".Length) }); var email = ConsumeSentMessage<SendEmail>(); Assert.Equal("Hibernating Rhinos - Trials Agent", email.ReplyToDisplay); }
As you can probably see, we have a structured way to send input to the system, and we can verify the output and the side affects (creating the trial, for example).
This leads to a system that can be easily tested, but doesn’t force us to spend too much time in the ceremony of tests.
Comments
Structuring the tests in a uniform way requires spending a negligible amount of time. The easier it is for a human to quickly read and interpret tests is a win, so I'm not sure what the downside of organizing the tests in this manner. If the argument is purely that it wastes time, that would seem to only apply if one were altering an existing code base of tests to follow the pattern. Again, the time cost is almost non-existent, so I don't understand the opposition.
well, phil talks about unit tests, while your example is not a unit test. hence, any comparisons may be invalid. personally i find your emphasis more useful for myself as well.
I don't really think Haack's structuring would require very much more time to implement. One thing I also like about it is it provides a guideline for a team to follow, otherwise everyone naturally ends up following their own approach and chaos follows.
"Spending a lot of extra time on the tests distract from creating real value, shippable software."
I don't see where Phil is spending lots of extra time? He's picked a convention for laying out his tests that others have found to be useful for maintenance and understanding.
If you treat them like a dumping ground and chuck things anywhere.. wait, did Ayende really post this?
I think structuring unit tests not only internally (following arrange, act and assert) but also externally (folder, subclasses or whatever method you use) is utterly important.
The larger the project grows and the longer the system is maintained the tests have to be maintaned too. If you apply a small set of conventions for internal and external structure it is very easy for developers who join the project later on or juniors to quickly feel comfortable in the code structure.The is actually a huge time saver and not a waste of time. You gain a lot of time because people don't need to filter and sort the tests again and again with tools help such as resharper.
I always treat test code like productive code. Because we spend time writing it so we also must spend time to improve, refactor and cleanup the test code. This applies for unit and developer acceptance tests...
Daniel
Daniel, That is exactly my point. Doing maintenance over tests is pure waste. They should be write once and stay there, pretty much. Working on tests means not working on other things.
If you do TDD, your code is going to naturally be structured against the source code because that's what the tests are driving: what appears in the source code.
There are different ways of organizing these tests, like one test class per production class, one class per method, one class per behavior, etc. What's important, though, is that the tests are driving and that they exist.
I tend to organize my tests by the use case. The test class name describes what is the use case and the actual test methods tell me what should happen. When the test fails I can see what part of the use case does not work.
public class When_order_is_received { public void Then_order_is_saved() public void Then_confirmation_email_is_sent() }
Irrelvant of what Phil is suggesting, I find structuring tests a critical sign of tests readability, which is very important as tests reflect the feature-set you are building.
I do use 3 levels of tests: 1) Unit tests: at the class level for Domain objects 2) Integration tests: usually involves a database or external service or file system. 3) UI tests: for asp.net and derived by feature tests.
You could argue about why units tests in the presence of integration tests, my answer is test time + fine grained feedback in TDD.
I probably missed something, but can someone please tell me what the Consume<T1, T2>() method does and where did it came from?
Ayende
Maybe this is me with my BDD hat on, but to me the tests are a representation of your requirements. Therefore if your requirements change, your tests change. Having well organised tests help with readability and maintainability in the same way as it does for production code.
Brilliant statement. Did anyone ever use them for any other purpose? I wasn't aware they had any other use.
Daniel, Instead of answering here, I scheduled a post about it, the early access link is here: http://ayende.com/blog/152993/testing-rhino-service-bus-applications?key=ee11d848dadc4923b8de78e6a7813888
Bob, To be fair, I know of a fair number of people who go to read the tests first, and use that as a source for information.
The most common excuse I've heard/used for not testing is that I can't figure out how to test. The second most common is the amount of time it takes to write and run tests. I think I would keep a pretty bare-bones organization of tests because of the latter case. It needs to be just enough that I don't waste time fiddling with the tests, no more.
That said, I'd probably organize the tests according the object being tested, that is, I'd have a class that has all the tests for a given class in the system. I don't think I'd go to the point of having nested classes for each test case. My main use of the unit tests is so I can get the thing to dump out a pass/fail in the console window using TestDriven.net. I find the GUI red/green/yellow test runners to be slow and somewhat visually distracting. The only reason I would want to keep tests together at the object level is so that I can just run the unit tests for the specific type in a batch as I'm wrecking....er.... tinkering with the object (I try to be aware of this because I'm frequently not using top of the line equipment).
I've used both test methods: using static classes to encapsulate state and more simple structures. It comes back to the idea that code is read a lot more than it's written. Tests provide a way for me to see how the code should work.
I agree that tests should stay in place for as long as the requirement driving the test is valid. I don't agree that tests won't (or shouldn't) ever change.
For me, it's a cost/benefit thing. If wrapping the tests in a static class makes them easier to read and understand, I'll do it. If not, I won't. Tests aren't just there for an automated runner to ensure things are working well, they're a diagnostic tool and documentation of the code. They're a way for me to communicate to another developer how the code should behave.
I will have to disagree with this one. I think that organizing anything -- whether they be tests, code, you name it -- it's a good thing as long as it doesn't distract from the main purpose.
whatever works i guess, personally i like Phill's approach on this. It make unit tests more organized and easier to look at when you have tons of unit test on one file
Sorry, but I also disagree. The approach Phil outlined is simple and effective, so why not.
Comment preview