Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 10 | Comments: 36

filter by tags archive

Don’t mock my integration tests

time to read 3 min | 533 words

Dror from TypeMock has managed to capture the essence of my post about unit tests vs. integration tests quite well:

Unit tests runs faster but integration tests are easier to write.

Unfortunately, he draws an incorrect conclusion out of that;

There is however another solution instead of declaring unit testing as hard to write - use an isolation framework which make writing unit test much easier.

And the answer to that is… no, you can’t do that. Doing something like that put you back in the onerous position of unit test, where you actually have to understand exactly what is going on and have to deal with that. With an integration test, you can assert directly on the end result, which is completely different than what I would have to do if I wanted to mock a part of the system. A typical integration test looks something like:

public class SelectNPlusOne : IScenario
{
    public void Execute(ISessionFactory factory)
    {
        using (var session = factory.OpenSession())
        using (var tx = session.BeginTransaction())
        {
            var blog = session.CreateCriteria(typeof(Blog))
                .SetMaxResults(1)
                .UniqueResult<Blog>();//1
            foreach (var post in blog.Posts)//2
            {
                Console.WriteLine(post.Comments.Count);// SELECT N
            }
            tx.Commit();
        }
    }
}


[Fact]
public void AllCallsToLoadCommentsAreIdentical()
{
    ExecuteScenarioInDifferentAppDomain<SelectNPlusOne>();
    var array = model.Sessions[0]
        .SessionStatements
        .ExcludeTransactions()
        .Skip(2)
        .ToArray();
    Assert.Equal(10, array.Length);
    foreach (StatementModel statementModel in array)
    {
        Assert.Equal(statementModel.RawSql, array[0].RawSql);
    }
}

This shows how we can assert on the actual end model of the system, without really trying to deal with what is actually going on. You cannot introducing any mocking to the mix without significantly hurting the clarity of the code.


Comments

Eli Lopian

I don't get your logic Oren.

Dror is talking about /Unit Tests/ and making them easier to write and you are giving an example of /Integration tests/ (are you comparing apples and oranges).

No one should mock integration tests, but you must agree that we should strive to make authoring unit tests easier.

Ayende Rahien

The comparison being made doesn't really hold, that is the problem. Saying that using TypeMock will make unit testing easier is false, because it is not that that makes unit tests harder than integrations

Eli Lopian

Great, Now we can discuss the real issue that was posted.

Do our frameworks make unit testing easier? Not compared to integration tests, but compare to writing unit tests without the tools.

Ayende Rahien

Not really. What I took objection to is the "There is however another solution instead" part, because that is not a solution to the problem at hand

Andrea Dallera

This looks to me like a low level spec. In an integration test I do care about what's been called and not what the results look like. In my scenario mocks are a necessary evil... they hurt readability real bad, i agree with that.

Dave

Like the pirates of the Caribbean I don't see TDD as a rule, but more like a guideline ;-) I do write my unittests before writing the code, but I do not follow the make one change rule. The unittests are written based on specifications. If the specifications say 1 + 1 = 3 than I must thread that as correct (even when I disagree).

Testing specifications is a hard subject to cover. As the architect I write most of the unittests. However several lead developers also contribute some unit tests to complement the specifications (like trowing exceptions and testing condidtions not part of the specification).

Integration tests are easier to write because you only have to setup the classes, most of the time you invoke some sort of method on a facade and evaluate the result. Why writing integration tests take so long. Most of the time it takes to write a integration test is not consumed by the test itself. I think 70-80% is spent to make sure that there is a consistent (persistent) data storage.

Integration tests are shallow tests, unit tests are in-depth tests. They both serve a different but equally important purpose.

Vagif Abilov

Oren, I think your statement "we can assert on the actual end model of the system, without really trying to deal with what is actually going on" exposes just part of the picture ("assert" part). But there is an "arrange" part that unfortunately often requires dealing with system internals. Setting up necessary data. Properly written unit tests usually don't need to go deep into data layers and if necessary fake/mock away anything that is not relevant to validate given functionality. On the other hand, properly written integration tests should be careful about not faking dependencies, and this is what makes them harder to write and set up.

Peter Morris

What I don't understand here is not really related to your post :-)

Why do you call this N+1?

If "O" is the number of selects then for each Blog ("O") you will have "N" selects not N+1. So shouldn't this be SelectN and not SelectNPlusOne, or am I missing something?

Ayende Rahien

It is the anti pattern name.

N is the number of posts per blog, to each we have to issue a select

1 is the first select, to bring the list of blogs

n+1

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Production postmortem: The case of the memory eater and high load - about one day from now
  2. Production postmortem: The case of the lying configuration file - 3 days from now
  3. Production postmortem: The industry at large - 4 days from now
  4. The insidious cost of allocations - 5 days from now
  5. Find the bug: The concurrent memory buster - 6 days from now

And 4 more posts are pending...

There are posts all the way to Sep 10, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    14 Aug 2015 - The case of the man in the middle
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats