Oren Eini

aka Ayende Rahien

Oren Eini

CEO of RavenDB

a NoSQL Open Source Document Database

Get in touch with me:

oren@ravendb.net +972 52-548-6969

Posts: 7,578
|
Comments: 51,203

Copyright ©️ Ayende Rahien 2004 — 2025

Privacy Policy · Terms
filter by tags archive
stack view grid view
  • architecture (608) rss
  • bugs (450) rss
  • challanges (123) rss
  • community (378) rss
  • databases (481) rss
  • design (894) rss
  • development (640) rss
  • hibernating-practices (71) rss
  • miscellaneous (592) rss
  • performance (397) rss
  • programming (1085) rss
  • raven (1445) rss
  • ravendb.net (529) rss
  • reviews (184) rss
  • 2025
    • May (10)
    • April (10)
    • March (10)
    • February (7)
    • January (12)
  • 2024
    • December (3)
    • November (2)
    • October (1)
    • September (3)
    • August (5)
    • July (10)
    • June (4)
    • May (6)
    • April (2)
    • March (8)
    • February (2)
    • January (14)
  • 2023
    • December (4)
    • October (4)
    • September (6)
    • August (12)
    • July (5)
    • June (15)
    • May (3)
    • April (11)
    • March (5)
    • February (5)
    • January (8)
  • 2022
    • December (5)
    • November (7)
    • October (7)
    • September (9)
    • August (10)
    • July (15)
    • June (12)
    • May (9)
    • April (14)
    • March (15)
    • February (13)
    • January (16)
  • 2021
    • December (23)
    • November (20)
    • October (16)
    • September (6)
    • August (16)
    • July (11)
    • June (16)
    • May (4)
    • April (10)
    • March (11)
    • February (15)
    • January (14)
  • 2020
    • December (10)
    • November (13)
    • October (15)
    • September (6)
    • August (9)
    • July (9)
    • June (17)
    • May (15)
    • April (14)
    • March (21)
    • February (16)
    • January (13)
  • 2019
    • December (17)
    • November (14)
    • October (16)
    • September (10)
    • August (8)
    • July (16)
    • June (11)
    • May (13)
    • April (18)
    • March (12)
    • February (19)
    • January (23)
  • 2018
    • December (15)
    • November (14)
    • October (19)
    • September (18)
    • August (23)
    • July (20)
    • June (20)
    • May (23)
    • April (15)
    • March (23)
    • February (19)
    • January (23)
  • 2017
    • December (21)
    • November (24)
    • October (22)
    • September (21)
    • August (23)
    • July (21)
    • June (24)
    • May (21)
    • April (21)
    • March (23)
    • February (20)
    • January (23)
  • 2016
    • December (17)
    • November (18)
    • October (22)
    • September (18)
    • August (23)
    • July (22)
    • June (17)
    • May (24)
    • April (16)
    • March (16)
    • February (21)
    • January (21)
  • 2015
    • December (5)
    • November (10)
    • October (9)
    • September (17)
    • August (20)
    • July (17)
    • June (4)
    • May (12)
    • April (9)
    • March (8)
    • February (25)
    • January (17)
  • 2014
    • December (22)
    • November (19)
    • October (21)
    • September (37)
    • August (24)
    • July (23)
    • June (13)
    • May (19)
    • April (24)
    • March (23)
    • February (21)
    • January (24)
  • 2013
    • December (23)
    • November (29)
    • October (27)
    • September (26)
    • August (24)
    • July (24)
    • June (23)
    • May (25)
    • April (26)
    • March (24)
    • February (24)
    • January (21)
  • 2012
    • December (19)
    • November (22)
    • October (27)
    • September (24)
    • August (30)
    • July (23)
    • June (25)
    • May (23)
    • April (25)
    • March (25)
    • February (28)
    • January (24)
  • 2011
    • December (17)
    • November (14)
    • October (24)
    • September (28)
    • August (27)
    • July (30)
    • June (19)
    • May (16)
    • April (30)
    • March (23)
    • February (11)
    • January (26)
  • 2010
    • December (29)
    • November (28)
    • October (35)
    • September (33)
    • August (44)
    • July (17)
    • June (20)
    • May (53)
    • April (29)
    • March (35)
    • February (33)
    • January (36)
  • 2009
    • December (37)
    • November (35)
    • October (53)
    • September (60)
    • August (66)
    • July (29)
    • June (24)
    • May (52)
    • April (63)
    • March (35)
    • February (53)
    • January (50)
  • 2008
    • December (58)
    • November (65)
    • October (46)
    • September (48)
    • August (96)
    • July (87)
    • June (45)
    • May (51)
    • April (52)
    • March (70)
    • February (43)
    • January (49)
  • 2007
    • December (100)
    • November (52)
    • October (109)
    • September (68)
    • August (80)
    • July (56)
    • June (150)
    • May (115)
    • April (73)
    • March (124)
    • February (102)
    • January (68)
  • 2006
    • December (95)
    • November (53)
    • October (120)
    • September (57)
    • August (88)
    • July (54)
    • June (103)
    • May (89)
    • April (84)
    • March (143)
    • February (78)
    • January (64)
  • 2005
    • December (70)
    • November (97)
    • October (91)
    • September (61)
    • August (74)
    • July (92)
    • June (100)
    • May (53)
    • April (42)
    • March (41)
    • February (84)
    • January (31)
  • 2004
    • December (49)
    • November (26)
    • October (26)
    • September (6)
    • April (10)
RavenDB - High-Performance NoSQL Document Database
  previous post next post  
May 18 2012

A bad test

time to read 1 min | 78 words

image

This is a bad test, because what it does is ensuring that something does not works. I just finished implementing the session.Advaned.Defer support, and this test got my attention by failing the build.

Bad test, you should be telling me when I broke something, not when I added new functionality.

Tweet Share Share 16 comments
Tags:
  • bugs

  previous post next post  

Comments

Daan
18 May 2012
09:14 AM
Daan

Not entirely true:

Your requirements are changed thus, your test doesn't meets the requirements anymore.

Indranil
18 May 2012
09:16 AM
Indranil

Ah but it gives 100% code coverage. :-P

Dalibor Carapic
18 May 2012
09:23 AM
Dalibor Carapic

I agree with Daan. If the test ensures that any client that uses your API will get a valid error when he tries something that is not supported then the test is just fine.

Rémi BOURGAREL
18 May 2012
09:31 AM
Rémi BOURGAREL

If you consider your doc as part of the API, then your Test is testing your doc.

maciejk
18 May 2012
09:49 AM
maciejk

Also, it can give great information to other programmers - they might be browsing tests to see how they're supposed to use this functionality and when they see this test they'll immediately know that it's not yet implemented.

Daniel Marbach
18 May 2012
10:14 AM
Daniel Marbach

How about that: You are changing the behavior of your API. If you'd follow test first approach you'd need to write a new test for your changes. Let it fail with the not supported exception, remove the assert throws from the other test and then implement the new feature. This is not a bad test. It is you not following the principles.

Daniel

Sergio
18 May 2012
11:28 AM
Sergio

It's a bad test, because you make more than one assertion in it, not because the reason you mentioned.

One test - one assertion. This way you know all failed assertions. In your test, all of them may be failing, or just the first one, you won't know.

Onofrio Panzarino
18 May 2012
11:40 AM
Onofrio Panzarino

I think that in the real world no one would be find useful that kind of test, neither very fanatic agilists. Making tests to succeed for real feature is enough a hard work.

Paul Stovell
18 May 2012
13:57 PM
Paul Stovell

I actually wrote a test like that against RavenDB.

A few versions ago there was a bug where Uri properties would cause Raven to think the data had always changed. So you could do this from memory:

var customer = session.Load<Customer>(); Assert.IsTrue(session.Advanced.HasChanges(customer));

I didn't want a broken test failing my build for a week while I waited for a new RavenDB build so I wrote it as a negative test. When I upgraded RavenDB and the test failed it was a nice reminder about the issue.

In some ways I think the negative test had /some/ value - if I ever wondered if a problem still existed I had some proof for it. But having a test 'fail' because something 'worked' did leave a strange feeling. I'm not sure what the solution for this should be. Maybe Assert.Inconclusive()?

Paul

Federico Lois
18 May 2012
14:08 PM
Federico Lois

As mentioned before the problem aint the test itself. It may be that it is hidden in an improper location without giving a semantic clue.

I do have a test file dedicated to "unsupported" or "defects" in third party components. When I update any of them (including RavenDB) and they are fixed I know right away and fix the workarounds in the code.

So I dont think it is a test, I believe it is a "note" to your team or yourself in the future where you are not actively thinking about it.

configurator
18 May 2012
15:11 PM
configurator

Parts of that test are actually quite important. shardedDocumentStore.Url can't possibly return a result that makes sense - if it returned (for example) the first shard's URL that would have been a bug. Same goes for DatabaseCommands and AsyncDatabaseCommands.

I think it's even important for GetLastWrittenEtag and Defer - because as long as you've not implemented it, an incorrect result can be far, far worse than a NotSupportedException.

Vadim K
18 May 2012
15:20 PM
Vadim K

This is why I love using machine.specifications for my testing framework. You can define a test without implementing it. This lets you know which features still need to be implemented.

Mike McG
18 May 2012
16:31 PM
Mike McG

Tests are designed to enforce the expectations made of the code. Those expectations can be set by interfaces, but can be supplemented by traditional documentation or other out-of-code communications.

In this case, if such communications have established that these methods are not supported, then the tests correctly enforce that (and are "good"). By failing when one of these methods is implemented, the developer is reminded that the expectations of the code have changed, and out-of-code communication artifacts must be updated to reflect that.

João P. Bragança
18 May 2012
16:58 PM
João P. Bragança

Crazy Idea: What about exposing

public static IEnumerable[Action[IDocumentStore]] OperationsNotSupported() {}

Then you could refactor without refuctoring.

Chris Wright
18 May 2012
23:37 PM
Chris Wright

There should have been separate tests for each of those functions, positioned where you would have naturally written the tests for the real, working functionality. That way, it wouldn't have shown up at the build server; it would have shown up as soon as you considered working on the feature that wasn't supported.

Glomming everything together means you have a submarine test, which, no matter what it's testing, is bad.

Had these tests been closer to where you were expecting, you wouldn't have thought to write this post, most likely.

[ICR]
20 May 2012
20:28 PM
[ICR]

Doesn't this suggest it should have been a NotImplementedException originally?

Comment preview

Comments have been closed on this topic.

Markdown formatting

ESC to close

Markdown turns plain text formatting into fancy HTML formatting.

Phrase Emphasis

*italic*   **bold**
_italic_   __bold__

Links

Inline:

An [example](http://url.com/ "Title")

Reference-style labels (titles are optional):

An [example][id]. Then, anywhere
else in the doc, define the link:
  [id]: http://example.com/  "Title"

Images

Inline (titles are optional):

![alt text](/path/img.jpg "Title")

Reference-style:

![alt text][id]
[id]: /url/to/img.jpg "Title"

Headers

Setext-style:

Header 1
========
Header 2
--------

atx-style (closing #'s are optional):

# Header 1 #
## Header 2 ##
###### Header 6

Lists

Ordered, without paragraphs:

1.  Foo
2.  Bar

Unordered, with paragraphs:

*   A list item.
    With multiple paragraphs.
*   Bar

You can nest them:

*   Abacus
    * answer
*   Bubbles
    1.  bunk
    2.  bupkis
        * BELITTLER
    3. burper
*   Cunning

Blockquotes

> Email-style angle brackets
> are used for blockquotes.
> > And, they can be nested.
> #### Headers in blockquotes
> 
> * You can quote a list.
> * Etc.

Horizontal Rules

Three or more dashes or asterisks:

---
* * *
- - - - 

Manual Line Breaks

End a line with two or more spaces:

Roses are red,   
Violets are blue.

Fenced Code Blocks

Code blocks delimited by 3 or more backticks or tildas:

```
This is a preformatted
code block
```

Header IDs

Set the id of headings with {#<id>} at end of heading line:

## My Heading {#myheading}

Tables

Fruit    |Color
---------|----------
Apples   |Red
Pears	 |Green
Bananas  |Yellow

Definition Lists

Term 1
: Definition 1
Term 2
: Definition 2

Footnotes

Body text with a footnote [^1]
[^1]: Footnote text here

Abbreviations

MDD <- will have title
*[MDD]: MarkdownDeep

 

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. Recording (16):
    29 May 2025 - RavenDB's Upcoming Optimizations Deep Dive
  2. Webinar (6):
    27 May 2025 - RavenDB's Upcoming Optimizations Deep Dive
  3. RavenDB News (2):
    02 May 2025 - May 2025
  4. Production Postmortem (52):
    07 Apr 2025 - The race condition in the interlock
  5. RavenDB (13):
    02 Apr 2025 - .NET Aspire integration
View all series

RECENT COMMENTS

  • What a massive presentation! As a person who spent some time with a db written in .NET I can strongly relate to some points. ...
    By Scooletz on Recording: RavenDB's Upcoming Optimizations Deep Dive
  • I’d love to learn your thoughts on SPANN https://arxiv.org/abs/2111.08566 that with centroids and keeping the posting lists s...
    By Scooletz on Comparing DiskANN in SQL Server & HNSW in RavenDB
  • Joel, The DiskANN paper talks about it being viable for more than a billion vectors datasets.  In such a scenario, it would ...
    By Oren Eini on Comparing DiskANN in SQL Server & HNSW in RavenDB
  • Do you know why they chose DiskANN? These things are usually about tradeoffs but it seems DiskANN is just worse in every way.
    By Joel on Comparing DiskANN in SQL Server & HNSW in RavenDB
  • Scooletz, Yes, we look at other stuff. Most of those are _not_ allowing you to build themselves incrementally nor are they...
    By Oren Eini on Scaling HNSW in RavenDB: Optimizing for inadequate hardware

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats
}