Ayende @ Rahien

Ayende Rahien commented on Scenario Driven Tests

Tue, 06 Oct 2009 09:52:15 GMT

Joao, NH Prof has something better, You can tell the application to save the output, and we can load it into the app as if it was a running app That make solving issues VERY easy

Ayende Rahien commented on Scenario Driven Tests

Fri, 02 Oct 2009 19:29:41 GMT

I would invoke the controllers directly

Gilligan commented on Scenario Driven Tests

Fri, 02 Oct 2009 16:03:37 GMT

i like the idea, especially if one can make them fast (by perhaps using an embedded database). But how would you plug into the Http Runtime if you were building a web app this way?

João P. Bragança commented on Scenario Driven Tests

Thu, 01 Oct 2009 03:13:00 GMT

Ayende, does nhprof have robust logging? Seems to me that if it did it would be easy to reproduce a scenario that blew up in a customer's face, then scenario test it and fix it on the next commit.

Frans Bouma commented on Scenario Driven Tests

Wed, 30 Sep 2009 09:29:23 GMT

"No, they don't, but I don't care for that. I care that all the scenarios that I know about are covered." I think that's essential info not determinable from the article. Essential to those who think using scenario tests (or unit tests in general for that matter) believe it will given them the idea of having solid code for scenarios they haven't thought of.

Ayende Rahien commented on Scenario Driven Tests

Wed, 30 Sep 2009 09:07:53 GMT

Frans, Yes, it was pretty obvious that you would know it, :-) it is what is called a leading question. Since I am talking about working implementations, I am thinking about testing in the same manner. I can never ensure that my application is 100% corrent. I think that I did injustice with the code samples, a scenario isn't just using the system API. A scenario is running the application. It is giving input, and then executing the app. > They don't prove that there are other scenarios, not tested, which also work No, they don't, but I don't care for that. I care that all the scenarios that I know about are covered.

Frans Bouma commented on Scenario Driven Tests

Wed, 30 Sep 2009 08:52:15 GMT

To clarify about halting problem remark: I'm talking about algorithms, not statements in a code file: tests test both: algorithm AND implementation, I only talk about algorithms, not implementation, as implementation has to be verified as it's a projection, algorithms have other ways to be tested, e.g. proving.

Frans Bouma commented on Scenario Driven Tests

Wed, 30 Sep 2009 08:23:09 GMT

"Are you familiar with the halting problem?" I'm stunned you even have to ask. ;) Of course I'm familiar with it. Though I dont see what it has to do with software correctness, as math proves software can be proven correct or not. "It is not _possible_ to get to 100%, I don't even try. My scenarios represent how the application works. They aren't executing code in specific ways, they exercise the entire application." Each scenario test takes 1 or more steps to complete. This is thus an order. Unless you're sure (thus proven) that this is the only order the application can execute the steps described in the scenario test, you can't be sure another scenario exists with for example a subset of the steps which for example fails. Each scenario test also uses data for input / drive the test. That's ok, but it's also the start set of a state, which evolves with each step. This too creates the possibility that there are other scenarios possible (which you didn't test) which make your app fail. This is important, read below. "When I get a bug, I have a test that expose it using an existing scenario or a new one. It is enough to show that the system works right, and to serve as regression tests." This is a fallacy: "the system works right". Your scenario tests prove one thing, and one thing only: that there ARE scenarios under which the application works as expected. They don't prove that there are other scenarios, not tested, which also work, simply because there's no evidence for that. If you think there IS evidence for that (so test for scenario X also proves the code works for scenario Y which isn't tested) please show me, as it seems to me unlikely there is. This isn't bad at all though. Having scenario tests which prove that the code at least works in scenario X (which is for example the likely scenario 99% of the users will follow) is great, as it proves the code is functional for the user, though with the restriction that the user has to follow the scenarios tested. That's the information that's overlooked by many TDD/BDD pundits who firmly believe, without any theoretical prove/argument, that a limited set of tests proves that their code is correct for situations not defined in tests. As long as a developer knows that the tests written aren't there to prove the code is correct but only that it works for the specific scenarios defined in tests, (unless other ways are used as well to prove further quality), it's OK. As I said before: the tests only show that what you designed works for at least 1 scenario, and that the code works for other scenarios as well is not based on the presence of any test whatsoever: it's based on either 1) luck, 2) crafmanship or 3) both.

Ayende Rahien commented on Scenario Driven Tests

Wed, 30 Sep 2009 05:49:43 GMT

Daniel, I knew why I was reluctant to write these two posts, because I expect this sort of comments.

Ayende Rahien commented on Scenario Driven Tests

Wed, 30 Sep 2009 05:47:36 GMT

Jamie, For something like this, I find it easier to not use mocking, I can write the fakes for the slow services manually, which will integrate into the test system and will have better perf characteristics. But I often will just use the real system, will all of its services

Daniel Cazzulino commented on Scenario Driven Tests

Wed, 30 Sep 2009 04:42:00 GMT

Integration/scenario tests are always a good thing to have around. They are completely orthogonal to TDD/unit testing, and far from a reason not to do them. My comments on this and the previous post, on why this doesn't work, at [www.clariusconsulting.net/.../...dowithoutTDD.aspx](http://www.clariusconsulting.net/blogs/kzu/archive/2009/09/29/AreyousmartenoughtodowithoutTDD.aspx)

Jamie Penney commented on Scenario Driven Tests

Wed, 30 Sep 2009 01:51:41 GMT

Do you just mock out slow stuff like file system access, db access, that sort of thing, so that the tests run faster? Do you find it difficult to design a full system that can be spun up very quickly for each new test? I am wondering because I am currently writing a small app in my spare time and am trying to adhere to current best practises. I've got unit tests for most of the functionality, but it is at a method level not a scenario level. As I already have a facade for my system so I my UI and scripting engine use the same code, would I be better off creating tests that set the system up through the facade and test it that way? Or should I perhaps even use both methods of testing?

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 17:40:39 GMT

Joe, Repeating asserts is far less common than repeating scenarios

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 17:39:49 GMT

Olav, I haven't run into that scenario yet, so I can't tell you. ExecuteScenario can be _very_ complicated, even without this. It need to setup the entire application and tear it down properly.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 17:38:21 GMT

Frans, Are you familiar with the halting problem? It is not _possible_ to get to 100%, I don't even try. My scenarios represent how the application works. They aren't executing code in specific ways, they exercise the entire application. When I get a bug, I have a test that expose it using an existing scenario or a new one. It is enough to show that the system works right, and to serve as regression tests.

Mike Murray commented on Scenario Driven Tests

Tue, 29 Sep 2009 17:12:45 GMT

Because he wants "to have a _clear_ separation between the test code and the tested scenario. They both can change independently." This was said by Ayende in this comment: [ayende.com/.../scenario-driven-tests.aspx#34839](http://ayende.com/Blog/archive/2009/09/29/scenario-driven-tests.aspx#34839)

Joe Gutierrez commented on Scenario Driven Tests

Tue, 29 Sep 2009 16:51:20 GMT

Why don't you also encode your assert within scenarios. ExecuteScenario Assert

Joe Gutierrez commented on Scenario Driven Tests

Tue, 29 Sep 2009 16:44:13 GMT

I can see that the scenarios can be extended. You can use them with an event driven design to code your application. ExecuteScenario

Olav commented on Scenario Driven Tests

Tue, 29 Sep 2009 14:54:41 GMT

I get that you don't do much mocking as this is about testing as much of the actual application as possible - i like it a lot. But when you do need to mock someting are there ever situations where you need to decide what to mock based on the senario beeing executed? If so, how do you handle this? Will the ExecuteSenario method not become very complex?

Will Smith commented on Scenario Driven Tests

Tue, 29 Sep 2009 14:48:09 GMT

I like how you can have one class with different test scenarios. Sometimes, BDD-style tests can feel like overkill (e.g. setting up the scenario in the pre-test phase for only one assertion). However, one thing I like about testing at a lower level is mitigating cyclomatic complexity. Testing a method allows me to focus on only the code paths for that method. Testing scenarios can grow significantly if you try to get full coverage. The ability to draw a line between components is invaluable in managing the number of tests necessary to get meaningful coverage. In general, I use "scenario" tests as acceptance tests. Our acceptance tests don't try to cover every scenario, only enough to demonstrate that the system does what the customer wants.

jdn commented on Scenario Driven Tests

Tue, 29 Sep 2009 14:45:42 GMT

"And no, I don’t mean something like BDD, where the code under test is being setup in the constructor or the context initialization method." I'm not sure that is a necessary condition for something being BDD. In any event, I'm testing scenarios the same way you are, in particular: "I don't care about methods, just about overall system behavior." I think that's spot on.

Frans Bouma commented on Scenario Driven Tests

Tue, 29 Sep 2009 13:27:37 GMT

"I don't try to test every single method. I test a _scenario_. The scenarios represent the things that I actually care about in my system. I don't care about methods, just about overall system behavior. " yes, but what kind of valuable information does it provide? It only provides that the code for _that_ scenario, executed in that order, works as expected. It says nothing about that scenario executed in a slightly different order, with slightly different data or a slightly different scenario even. I.o.w.: it's good for testing that scenario, but you can't use it for determining if your code will work for other scenarios than the ones you've tested. If you don't use other means to cover that, it's actually simply testing 'a few' scenarios out of a range of infinite scenarios and declaring, based on those, that the code will work. That the code works in other scenarios (otherwise you'd be swamped with support calls ;)) is thus not the result of this kind of testing (as the tests have no value for these non-tested scenarios) but the result of something else, e.g. years of experience, re-usage of proven solid code, implementations of well-designed algorithms etc. etc.

Thomas Andersson commented on Scenario Driven Tests

Tue, 29 Sep 2009 12:02:48 GMT

Thomas, TDD is the approach/work-method where you follow the Red-Green-Refactor cycle. Ayende tend to not use this way to work, but rather do small spikes and try out different designs and then add the scenario-tests. Thus he's not using TDD when doing these tests.

Thomas Eyde commented on Scenario Driven Tests

Tue, 29 Sep 2009 11:17:55 GMT

Ayende, you know best what you don't do :-) I agree with your approach, which is similar to what I try to achieve. I usually say my tests are BDD-inspired, but not actually BDD. I don't think we need yet another name, so I prefer to think of my approach as a variation of TDD, but still TDD. What is it with your approach that breaks with TDD?

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:32:03 GMT

Frans, I don't try to test every single method. I test a _scenario_. The scenarios represent the things that I actually care about in my system. I don't care about methods, just about overall system behavior.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:30:30 GMT

Torkel, In NH Prof, those tests will stop just short of the UI. I am usually asserting on the View Model. I don't bother with classification.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:29:20 GMT

Andrey, Not all scenarios are as simple as that. And I want to have a _clear_ separation between the test code and the tested scenario. They both can change independently.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:28:15 GMT

David, Yes, it doesn't really make sense to try to do this in a library, a library does only a single thing. An application is much more complex and require a different strategy.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:26:48 GMT

Adam, I setup the env. in the ExecuteScenario, yes. If mocking is done, it will be done there, but I don't tend to use mocking much there.

Ayende Rahien commented on Scenario Driven Tests

Tue, 29 Sep 2009 10:24:59 GMT

Thomas, It is not TDD, because TDD is an approach. I am using tests, but I don't use TDD.