Ayende @ Rahien

Refunds available at head office

Don’t mock my integration tests

Dror from TypeMock has managed to capture the essence of my post about unit tests vs. integration tests quite well:

Unit tests runs faster but integration tests are easier to write.

Unfortunately, he draws an incorrect conclusion out of that;

There is however another solution instead of declaring unit testing as hard to write - use an isolation framework which make writing unit test much easier.

And the answer to that is… no, you can’t do that. Doing something like that put you back in the onerous position of unit test, where you actually have to understand exactly what is going on and have to deal with that. With an integration test, you can assert directly on the end result, which is completely different than what I would have to do if I wanted to mock a part of the system. A typical integration test looks something like:

public class SelectNPlusOne : IScenario
{
    public void Execute(ISessionFactory factory)
    {
        using (var session = factory.OpenSession())
        using (var tx = session.BeginTransaction())
        {
            var blog = session.CreateCriteria(typeof(Blog))
                .SetMaxResults(1)
                .UniqueResult<Blog>();//1
            foreach (var post in blog.Posts)//2
            {
                Console.WriteLine(post.Comments.Count);// SELECT N
            }
            tx.Commit();
        }
    }
}


[Fact]
public void AllCallsToLoadCommentsAreIdentical()
{
    ExecuteScenarioInDifferentAppDomain<SelectNPlusOne>();
    var array = model.Sessions[0]
        .SessionStatements
        .ExcludeTransactions()
        .Skip(2)
        .ToArray();
    Assert.Equal(10, array.Length);
    foreach (StatementModel statementModel in array)
    {
        Assert.Equal(statementModel.RawSql, array[0].RawSql);
    }
}

This shows how we can assert on the actual end model of the system, without really trying to deal with what is actually going on. You cannot introducing any mocking to the mix without significantly hurting the clarity of the code.

Comments

Eli Lopian
05/13/2009 05:49 AM by
Eli Lopian

I don't get your logic Oren.

Dror is talking about /Unit Tests/ and making them easier to write and you are giving an example of /Integration tests/ (are you comparing apples and oranges).

No one should mock integration tests, but you must agree that we should strive to make authoring unit tests easier.

Ayende Rahien
05/13/2009 05:52 AM by
Ayende Rahien

The comparison being made doesn't really hold, that is the problem. Saying that using TypeMock will make unit testing easier is false, because it is not that that makes unit tests harder than integrations

Eli Lopian
05/13/2009 05:57 AM by
Eli Lopian

Great, Now we can discuss the real issue that was posted.

Do our frameworks make unit testing easier? Not compared to integration tests, but compare to writing unit tests without the tools.

Ayende Rahien
05/13/2009 06:14 AM by
Ayende Rahien

Not really. What I took objection to is the "There is however another solution instead" part, because that is not a solution to the problem at hand

Andrea Dallera
05/13/2009 06:16 AM by
Andrea Dallera

This looks to me like a low level spec. In an integration test I do care about what's been called and not what the results look like. In my scenario mocks are a necessary evil... they hurt readability real bad, i agree with that.

Dave
05/13/2009 07:04 AM by
Dave

Like the pirates of the Caribbean I don't see TDD as a rule, but more like a guideline ;-) I do write my unittests before writing the code, but I do not follow the make one change rule. The unittests are written based on specifications. If the specifications say 1 + 1 = 3 than I must thread that as correct (even when I disagree).

Testing specifications is a hard subject to cover. As the architect I write most of the unittests. However several lead developers also contribute some unit tests to complement the specifications (like trowing exceptions and testing condidtions not part of the specification).

Integration tests are easier to write because you only have to setup the classes, most of the time you invoke some sort of method on a facade and evaluate the result. Why writing integration tests take so long. Most of the time it takes to write a integration test is not consumed by the test itself. I think 70-80% is spent to make sure that there is a consistent (persistent) data storage.

Integration tests are shallow tests, unit tests are in-depth tests. They both serve a different but equally important purpose.

Vagif Abilov
05/13/2009 03:39 PM by
Vagif Abilov

Oren, I think your statement "we can assert on the actual end model of the system, without really trying to deal with what is actually going on" exposes just part of the picture ("assert" part). But there is an "arrange" part that unfortunately often requires dealing with system internals. Setting up necessary data. Properly written unit tests usually don't need to go deep into data layers and if necessary fake/mock away anything that is not relevant to validate given functionality. On the other hand, properly written integration tests should be careful about not faking dependencies, and this is what makes them harder to write and set up.

Peter Morris
05/14/2009 01:32 PM by
Peter Morris

What I don't understand here is not really related to your post :-)

Why do you call this N+1?

If "O" is the number of selects then for each Blog ("O") you will have "N" selects not N+1. So shouldn't this be SelectN and not SelectNPlusOne, or am I missing something?

Ayende Rahien
05/14/2009 01:35 PM by
Ayende Rahien

It is the anti pattern name.

N is the number of posts per blog, to each we have to issue a select

1 is the first select, to bring the list of blogs

n+1

Comments have been closed on this topic.