Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,128 | Comments: 45,546

filter by tags archive



Not entirely true:

Your requirements are changed thus, your test doesn't meets the requirements anymore.


Ah but it gives 100% code coverage. :-P

Dalibor Carapic

I agree with Daan. If the test ensures that any client that uses your API will get a valid error when he tries something that is not supported then the test is just fine.


If you consider your doc as part of the API, then your Test is testing your doc.


Also, it can give great information to other programmers - they might be browsing tests to see how they're supposed to use this functionality and when they see this test they'll immediately know that it's not yet implemented.

Daniel Marbach

How about that: You are changing the behavior of your API. If you'd follow test first approach you'd need to write a new test for your changes. Let it fail with the not supported exception, remove the assert throws from the other test and then implement the new feature. This is not a bad test. It is you not following the principles.



It's a bad test, because you make more than one assertion in it, not because the reason you mentioned.

One test - one assertion. This way you know all failed assertions. In your test, all of them may be failing, or just the first one, you won't know.

Onofrio Panzarino

I think that in the real world no one would be find useful that kind of test, neither very fanatic agilists. Making tests to succeed for real feature is enough a hard work.

Paul Stovell

I actually wrote a test like that against RavenDB.

A few versions ago there was a bug where Uri properties would cause Raven to think the data had always changed. So you could do this from memory:

var customer = session.Load(); Assert.IsTrue(session.Advanced.HasChanges(customer));

I didn't want a broken test failing my build for a week while I waited for a new RavenDB build so I wrote it as a negative test. When I upgraded RavenDB and the test failed it was a nice reminder about the issue.

In some ways I think the negative test had /some/ value - if I ever wondered if a problem still existed I had some proof for it. But having a test 'fail' because something 'worked' did leave a strange feeling. I'm not sure what the solution for this should be. Maybe Assert.Inconclusive()?


Federico Lois

As mentioned before the problem aint the test itself. It may be that it is hidden in an improper location without giving a semantic clue.

I do have a test file dedicated to "unsupported" or "defects" in third party components. When I update any of them (including RavenDB) and they are fixed I know right away and fix the workarounds in the code.

So I dont think it is a test, I believe it is a "note" to your team or yourself in the future where you are not actively thinking about it.


Parts of that test are actually quite important. shardedDocumentStore.Url can't possibly return a result that makes sense - if it returned (for example) the first shard's URL that would have been a bug. Same goes for DatabaseCommands and AsyncDatabaseCommands.

I think it's even important for GetLastWrittenEtag and Defer - because as long as you've not implemented it, an incorrect result can be far, far worse than a NotSupportedException.

Vadim K

This is why I love using machine.specifications for my testing framework. You can define a test without implementing it. This lets you know which features still need to be implemented.

Mike McG

Tests are designed to enforce the expectations made of the code. Those expectations can be set by interfaces, but can be supplemented by traditional documentation or other out-of-code communications.

In this case, if such communications have established that these methods are not supported, then the tests correctly enforce that (and are "good"). By failing when one of these methods is implemented, the developer is reminded that the expectations of the code have changed, and out-of-code communication artifacts must be updated to reflect that.

João P. Bragança

Crazy Idea: What about exposing

public static IEnumerable[Action[IDocumentStore]] OperationsNotSupported() {}

Then you could refactor without refuctoring.

Chris Wright

There should have been separate tests for each of those functions, positioned where you would have naturally written the tests for the real, working functionality. That way, it wouldn't have shown up at the build server; it would have shown up as soon as you considered working on the feature that wasn't supported.

Glomming everything together means you have a submarine test, which, no matter what it's testing, is bad.

Had these tests been closer to where you were expecting, you wouldn't have thought to write this post, most likely.


Doesn't this suggest it should have been a NotImplementedException originally?

Comment preview

Comments have been closed on this topic.


  1. The low level interview question - 13 hours from now
  2. The worker pattern - 4 days from now

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats