Actual scenario testing with Raven

Yesterday I posted about doing scenario testing with Raven, and I showed the concept of what i am doing. This time, I wanted to show what I am actually talking about, and how this is implemented. Here are the current scenarios for Raven.

Each scenario is looks something like this (showing PutAndGetDocument here):

And the second request:

The scenarios are being picked up using:

public class AllScenariosWithoutExplicitScenario
{
    [Theory]
    [PropertyData("ScenariosWithoutExplicitScenario")]
    public void Execute(string file)

        new Scenario(Path.Combine(ScenariosPath, file+".saz")).Execute();
    }

    public static string ScenariosPath
    {
        get
        {
            return Directory.Exists(@"..\..\bin") // running in VS
                       ? @"..\..\Scenarios" : @"..\Raven.Scenarios\Scenarios";
        }
    }

    public static IEnumerable<object[]> ScenariosWithoutExplicitScenario
    {
        get
        {
            foreach (var file in Directory.GetFiles(ScenariosPath,"*.saz"))
            {
                if (typeof(Scenario).Assembly.GetType("Raven.Scenarios." +
                          Path.GetFileNameWithoutExtension(file) +"Scenario") != null)
                    continue;
                yield return new object[] {Path.GetFileNameWithoutExtension(file)};
            };
        }
    }
}

There are two reasons why I am ignoring explicit scenarios. Adding a class for a specific scenario allows me to run the scenario in the debugger, and also allow me to selectively skip certain scenarios if I need to.

Scenario.Execute is fairly involved, it parse the Fiddler’s saz file, build appropriate request and compare to the expect response, it is also smart enough to handle changing things like ETags and pass them along.

The end result is that I can very easily add new scenarios as I get new features to that requires tests.

Comments

03/05/2010 01:18 AM by

Ajai Shankar

I think I've also had some success with the scenario based approach (state black box)

We had complex mortgage lead distribution rules which were nicely captured in a scenario xml file.

Even built a simple Scenario UI where we could set up current and expected state after the distribution code ran.

My MBUnit test looked like this:

DataSet data; // my mock DB without NH :-)

[Factory]

public string[] ScenarioFile()

{

string dir = ConfigurationSettings.AppSettings[SCENARIO_DIR];

return Directory.GetFiles(dir, "*.xml");

}

[CombinatorialTest]

public void TestOneScenario([UsingFactories("ScenarioFile")] string scenarioFile)

{

data.Clear();

data.ReadXml(scenarioFile, XmlReadMode.IgnoreSchema);

LeadDistributorMock mock = new LeadDistributorMock(DB, data);

Assert.IsTrue(mock.RunAndVerifyDistribution());

}

And some files were:

DailyMax.xml

DistributionGroupPriority.xml

FreeLead_AggregatorPricing.xml

FreeLead_IOBudget.xml

03/05/2010 06:50 AM by

Andrey Shchekin

Nice. Seems very similar to RowTests/Factories people did with MbUnit a lot of time ago. I wouldn't say this approach is specific to scenarios, with other kinds of data in files it can be used for unit-testing as well.

Btw why "running in VS" hack is required? I generally just put required files into embedded resources.

03/05/2010 07:54 AM by

Ayende Rahien

Andrey,

Because I want an easy way to be able to debug a single scenario, rather than executing all of them in one shot.

03/05/2010 10:32 AM by

Imran

You could also use something like watiN http://watin.sourceforge.net/.

03/05/2010 10:35 AM by

Ayende Rahien

Imran,

No, I could not.

Different purposes all together

03/05/2010 12:37 PM by

Imran

yeh just realised that you don't actually have have a web form. I saw 'get' methods and assumed this.

03/11/2010 09:20 PM by

Jay Walker

Ayende,

i'm interested in how you are parsing the .saz (zip) file. FiddlerCore doesn't include this ability. I'm interested if you are unzipping and then using FiddlerCore. I'd like to take advantage of some Fiddler functions to give timing information across a graph of requests (html + associated images for instance).

Scenario.Execute is fairly involved, it parse the Fiddler’s saz file, build appropriate request and compare to the expect response, it is also smart enough to handle changing things like ETags and pass them along.

03/12/2010 08:08 AM by

Ayende Rahien

Jay,

I wasn't actually aware that there WAS a fiddler core :-(

I unzipped the files manually, read the stored values and acted upon them, nothing smart or anything like it, I am afraid.

Comments have been closed on this topic.

Ayende @ Rahien

Unnatural acts on source code

Actual scenario testing with Raven

Comments

Main feed
Comments feed