Ayende @ Rahien

Hi!
My name is Ayende Rahien
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

@

Posts: 5,947 | Comments: 44,540

filter by tags archive

Testing Rhino Service Bus applications


One of the really nice things about Rhino Service Bus applications is that we have created a structured way to handle inputs and outputs. You have messages coming in and out, as well as the endpoint local state to deal with. You don’t have to worry about how to deal with external integration points, because those are already going over messages.

And when you have basic input/output figured out, you are pretty much done.

For example, let us see the code that handles extending trail licenses in our ordering system:

public class ExtendTrialLicenseConsumer : ConsumerOf<ExtendTrialLicense>
{
    public IDocumentSession Session { get; set; }
    public IServiceBus Bus { get; set; }

    public void Consume(ExtendTrialLicense message)
    {
        var productId = message.ProductId ?? "products/" + message.Profile;
        var trial = Session.Query<Trial>()
            .Where(x => x.Email == message.Email && x.ProductId == productId)
            .FirstOrDefault();

        if (trial == null)
            return;
        
        trial.EndsAt = DateTime.Today.AddDays(message.Days);
        Bus.Send(new NewTrial
        {
            ProductId = productId,
            Email = trial.Email,
            Company = trial.Company,
            FullName = trial.Name,
            TrackingId = trial.TrackingId
        });
    }
}

How do we test something like this? As it turns out, quite easily:

public class TrailTesting : ConsumersTests
{
    protected override void PrepareData(IDocumentSession session)
    {
        session.Store(new Trial
        {
            Email = "you@there.gov",
            EndsAt = DateTime.Today,
            ProductId = "products/nhprof"
        });
    }

    [Fact]
    public void Will_update_trial_date()
    {
        Consume<ExtendTrialLicenseConsumer, ExtendTrialLicense>(new ExtendTrialLicense
        {
            ProductId = "products/nhprof",
            Days = 30,
            Email = "you@there.gov",
        });

        using (var session = documentStore.OpenSession())
        {
            var trial = session.Load<Trial>(1);
            Assert.Equal(DateTime.Today.AddDays(30), trial.EndsAt);
        }
    }

    // more tests here
}

All the magic happens in the base class, though:

public abstract class ConsumersTests : IDisposable
{
    protected IDocumentStore documentStore;
    private IServiceBus Bus = new FakeBus();

    protected ConsumersTests()
    {
        documentStore = new EmbeddableDocumentStore
        {
            RunInMemory = true,
            Conventions =
                {
                    DefaultQueryingConsistency = ConsistencyOptions.QueryYourWrites
                }
        }.Initialize();

        IndexCreation.CreateIndexes(typeof(Products_Stats).Assembly, documentStore);

        Products.Create(documentStore);
        using (var session = documentStore.OpenSession())
        {
            PrepareData(session);
            session.SaveChanges();
        }
    }

    protected T ConsumeSentMessage<T>()
    {
        var fakeBus = ((FakeBus)Bus);
        object o = fakeBus.Messages.Where(x => x.GetType() == typeof(T)).First();

        fakeBus.Messages.Remove(o);
        return (T) o;
    }

    protected void Consume<TConsumer, TMsg>(TMsg msg)
        where TConsumer : ConsumerOf<TMsg>, new()
    {
        var foo = new TConsumer();

        using (var documentSession = documentStore.OpenSession())
        {
            Set(foo, documentSession);
            Set(foo, Bus);
            Set(foo, documentStore);

            foo.Consume(msg);

            documentSession.SaveChanges();
        }
    }

    private void Set<T,TValue>(T foo, TValue value)
    {
        PropertyInfo firstOrDefault = typeof(T).GetProperties().FirstOrDefault(x=>x.PropertyType==typeof(TValue));
        if (firstOrDefault == null) return;
        firstOrDefault.SetValue(foo, value, null);
    }

    protected virtual void PrepareData(IDocumentSession session)
    {
    }

    public void Dispose()
    {
        documentStore.Dispose();
    }
}

And here are the relevant details for the FakeBus implementation:

public class FakeBus : IServiceBus
{
    public List<object>  Messages = new List<object>();

    public void Send(params object[] messages)
    {
        Messages.AddRange(messages);
    }
}

Now, admittedly, this is a fairly raw approach and we can probably do better. This is basically hand crafted auto mocking for consumers, and I don’t like the Consume<TConsumer,TMsg>() syntax very much. But it works, it is simple and it doesn’t really gets in the way.

I won’t say it is the way to go about it, but it is certainly easier than many other efforts that I have seen. We just need to handle the inputs & outputs and have a way to look at the local state, and you are pretty much done.

Rhino Service Bus & RavenDB integration


One of the interesting things about Rhino Service Bus is that I explicitly designed it to work nicely with Unit of Work style data access libraries. When I did that, I worked mainly with NHibernate, but it turns out that this is really easy to integrate RavenDB as well, all you need is the following message module:

public class RavenDbMessageModule : IMessageModule
{
    private readonly IDocumentStore documentStore;

    [ThreadStatic]
    private static IDocumentSession currentSession;

    public static IDocumentSession CurrentSession
    {
        get { return currentSession; }
    }

    public RavenDbMessageModule(IDocumentStore documentStore)
    {
        this.documentStore = documentStore;
    }

    public void Init(ITransport transport, IServiceBus serviceBus)
    {
        transport.MessageArrived += TransportOnMessageArrived;
        transport.MessageProcessingCompleted += TransportOnMessageProcessingCompleted;
    }

    public void Stop(ITransport transport, IServiceBus serviceBus)
    {
        transport.MessageArrived -= TransportOnMessageArrived;
        transport.MessageProcessingCompleted -= TransportOnMessageProcessingCompleted;
    }

    private static void TransportOnMessageProcessingCompleted(CurrentMessageInformation currentMessageInformation, Exception exception)
    {
        if (currentSession != null)
        {
            if (exception == null)
                currentSession.SaveChanges();
            currentSession.Dispose();
        }
        currentSession = null;
    }

    private bool TransportOnMessageArrived(CurrentMessageInformation currentMessageInformation)
    {
        if (currentSession == null)
            currentSession = documentStore.OpenSession();
        return false;
    }
}

This is fairly simple. Register to the message arrive and message processing completed events. When a message arrive, create a new session for the consumers. When the message processing is completed, and there hasn’t been any error, we call SaveChanges, and then dispose.

The rest of it is pretty simple as well, we need to provide a BootStrapper:

public class BootStrapper : CastleBootStrapper
{
    IDocumentStore store;

    protected override void ConfigureContainer()
    {
        store = new DocumentStore
        {
            ConnectionStringName = "RavenDB"
        }.Initialize();

        IndexCreation.CreateIndexes(typeof(BootStrapper).Assembly, store);

        Container.Register(
            Component.For<IDocumentStore>()
                .Instance(store),
            Component.For<IMessageModule>()
                .ImplementedBy<RavenDbMessageModule>(),
            Component.For<IDocumentSession>()
                .UsingFactoryMethod(() => RavenDbMessageModule.CurrentSession)
            );

        base.ConfigureContainer();
    }
}

Which basically simply need to create the document store and expose it to the container. We get the document session from the current one (the one managed by the module).

All in all, it is quite a thing, and it takes very little time / complexity to setup.

Setting up a Rhino Service Bus ApplicationPart II–One way bus


One really nice feature of Rhino Service Bus is the notion of the one way bus. What is that? It is a miniature implementation of the bus that supports only sending messages, not receiving them. In what world is this useful?

It turn out, it quite a few. One way bus is usually used for web apps that just send commands / events to another system, and have no need to interact with the bus other than that, or for command line tools that just send a message, etc. The advantage of the one way bus is that you don’t need your own endpoint to use it, you can just start it, send some messages, and go away.

Here is how you set it up. As usual, we start from the configuration (this assumes you have the Raven.ServiceBus.Castle nuget package):

<?xml version="1.0"?>
<configuration>
  <configSections>
    <section name="rhino.esb" type="Rhino.ServiceBus.Config.BusConfigurationSection, Rhino.ServiceBus"/>
  </configSections>
  <rhino.esb>
    <messages>
      <add name="HibernatingRhinos.Orders.Backend.Messages"
           endpoint="msmq://localhost/Orders.Backend"/>
    </messages>
  </rhino.esb>
</configuration>

You might note that we don’t have any bus/endpoint configuration, only the list of message owners.

Now, the next step is just to create the actual one way bus:

var container = new WindsorContainer();
new OnewayRhinoServiceBusConfiguration()
    .UseCastleWindsor(container)
    .Configure();

var onewayBus = container.Resolve<IOnewayBus>();
onewayBus.Send(new TestMsg
{
    Name = “ayende”
});

And that is all…

Setting up a Rhino Service Bus ApplicationPart I


Part of what we are currently doing at Hibernating Rhinos is work on some internal applications. In particular, we slowly update our existing applications to do two things:

  • Make sure that we are running a maintainable software package.
  • Dog food our own stuff in real world scenarios.

In this case, I want to talk about Rhino Service Bus. Corey has been maintaining the project and I haven’t really followed very closely on what was going on. Today we started doing major work on our order processing backend system, so I decided that we might as well ditch the 2.5+ years old version of Rhino Service Bus that we were running and get working with the latest.

I was so lazy about that that I actually didn’t bother to get the sources, and just got the nuget package.

Install-Package Rhino.ServiceBus.Castle

Install-Package Raven.ServiceBus.Host

That got me half way there. No need to worry about adding references, etc, what an awesome idea.

Next, I had to deal with the configuration. Here is how you configure a Rhino Service Bus application (app.config):

<configuration>
    <configSections>
        <section name="rhino.esb"
                         type="Rhino.ServiceBus.Config.BusConfigurationSection, Rhino.ServiceBus"/>
    </configSections>
    <rhino.esb>
        <bus
            threadCount="1"
            numberOfRetries="5"
            endpoint="msmq://localhost/Orders.Backend"
                />
        <messages>
            <add name="HibernatingRhinos.Orders.Backend.Messages"
                     endpoint="msmq://localhost/Orders.Backend"/>
        </messages>
    </rhino.esb>
</configuration>

As you can see, it is fairly simple. Specify the endpoint, and the ownerships of the messages, and you are pretty much done.

The next step is to setup the actual application. In order to do that, we need to define a boot strapper. Think about it like a constructor, but for the entire application. The minimal boot strapper needed is:

public class BootStrapper : CastleBootStrapper
{
    
}

Again, it doesn’t really get any simpler.

The next step is somewhat of a pain. A Rhino Service Bus project is usually a class library (dll). So if we want to run it, we need to host it somehwere. RSB comes with Rhino.ServiceBus.Host, but the command line interface sucks. You need to do this to get it running:

Rhino.ServiceBus.Host.exe /Action:Debug /Name:HibernatingRhinos.Orders.Backend /Assembly:.\HibernatingRhinos.Orders.Backend.dll

And honestly, who ever thought about making it so complicated? Give me something to make this simpler, for crying out loud. Moron developer who no consideration of actual system usability.

Oh, yeah, that would be me. Maybe I ought to fix that.

Anyway, that is pretty much everything you need to do to get RSB up & running. Here is a sample test consumer:

public class TestConsumer : ConsumerOf<TestMsg>
{
    public void Consume(TestMsg message)
    {
        Console.WriteLine(message.Name);
    }
}

And that is enough for now. See you in the next post…

Public Service Announcement: Git master repositories for the Rhino Tools projects


There have been some changes, and it seems that it is hard to track them. Here are where you can find the master repositories for the rhino tools projects:

Handling production errors in a messaging environment


So, today I got the first L2S Prof order. As you can imagine, I was pretty excited about that. However, it turned out that I had actually missed something when I built the backend for handling L2S Prof ordering. The details about what actually went wrong aren’t important (and are embarrassing).

But I logged into the server and checked out what was going on:

image

One of the major design criteria that I had with Rhino Service Bus is that it should be dead easy to handle production. As you can see in the screen shots, I have two messages of interest here, the first one is the actual message, and the second is the error information, which include the full stack trace. Using that, it was a piece of cake to isolate the problem, do the head slapping moment, and deploy a new version out. Once that was done, all I had to do is to move the message back to the processing queue, and I was done.

Just to give you an idea, here is how it looks like on my timeline:

image

I guarantee you that the customer in question didn’t have any idea that there was something wrong with his order processing.

I am loving it.

Using a service bus for queries


image Yep, another forum question. Unfortunately, in this case all I have is the title. Even more unfortunately, I already used the stripper metaphor before.

There are some questions that I am really not sure how to answer, because there are several underlying premises that are flat out wrong in the mere asking of the question.

“Can you design a square wheel carriage?” is a good example of that, and using a service bus for queries is another.

The short answer is that you don’t do that.

The longer answer is that you still don’t do that, but also explains why the question itself is wrong. One of the things that goes along with a service bus is the concept of services.

image Services are autonomous.

Does this ring a bell?

You don’t query a service for its state, because that would violate the autonomy tenant.

But I need the users data from the Personalization service to show the home page, I can hear you say. Well, sure, but you don’t perform queries across a service boundary.

 

Notice the terminology here. You don’t perform queries across a service boundary.

But you can perform queries inside a service boundary. The image on the right shows one such example of that.

We have several services in a single application, they communicate between services using a service bus.

But a service isn’t just something that is running in a server somewhere. The personalization service also have user interface, business logic that needs to run on the UI, etc.

That isn’t just some other part of the application that is accessing the personalization service operations. It is a part of the personalization service.

And inside a service boundary, there are no limitation to how you get the data you need to perform some operation.

You can perform a query using whatever method you like (a synchronous web service call, hitting the service database, using local state).

Personally, I would either access the data store directly (which usually means querying the service database) or use local state. I spoke about building a system where all queries are handled using local state here.

On PSake


James Kovacks introduced psake ( a power shell based build system )over a year ago, and at the time, I gave it a glance and decided that it was interesting, but not worth further investigation.

This weekend, as I was restructuring my Rhino Tools project, I realized that I need to touch the build system as well. The Rhino Tools build system has been through several projects, and was originally ported from Hibernate. It is NAnt based, complex, and can do just about everything that you want expect be easily understandable.

It became clear to me very quickly that it ain’t going to be easy to change the way it works, nor would it be easy to modify that to reflect the new structure. There are other issues with complex build systems, they tend to create zones of “there be dragons”, where only the initiated go, and even they go with trepidation. I decided to take advantage of the changes that I am already making to get a simpler build system.

I had a couple of options open to me: Rake and Bake.

Bake seemed natural, until I remember that no one touched it in a year or two. Beside, I can only stretch NIH so far :-). And while I know that people rave about rake, I did not want to introduce a Ruby dependency on my build system. I know that it was an annoyance when I had to build Fluent NHibernate.

One thing that I knew that I am not willing to go back to was editing XML, so I started looking at other build systems, ending up running into PSake.

There are a few interesting things that reading about it brought to mind. First, NAnt doesn’t cut it anymore. It can’t build WPF applications nor handle multi targeting well. Second, I am already managing the compilation part of the build using MSBuild, thanks to Visual Studio.

That leave the build system with executing msbuild, setting up directories, executing tests, running post build tools, etc.

PSake handles those well, since the execution environment is the command line. The syntax is nice, just enough to specify tasks and dependencies, but everything else is just pure command line. The following is Rhino Mocks build script, using PSake:

properties { 
  $base_dir  = resolve-path .
  $lib_dir = "$base_dir\SharedLibs"
  $build_dir = "$base_dir\build" 
  $buildartifacts_dir = "$build_dir\" 
  $sln_file = "$base_dir\Rhino.Mocks-vs2008.sln" 
  $version = "3.6.0.0"
  $tools_dir = "$base_dir\Tools"
  $release_dir = "$base_dir\Release"
} 

task default -depends Release

task Clean { 
  remove-item -force -recurse $buildartifacts_dir -ErrorAction SilentlyContinue 
  remove-item -force -recurse $release_dir -ErrorAction SilentlyContinue 
} 

task Init -depends Clean { 
    . .\psake_ext.ps1
    Generate-Assembly-Info `
        -file "$base_dir\Rhino.Mocks\Properties\AssemblyInfo.cs" `
        -title "Rhino Mocks $version" `
        -description "Mocking Framework for .NET" `
        -company "Hibernating Rhinos" `
        -product "Rhino Mocks $version" `
        -version $version `
        -copyright "Hibernating Rhinos & Ayende Rahien 2004 - 2009"
        
    Generate-Assembly-Info `
        -file "$base_dir\Rhino.Mocks.Tests\Properties\AssemblyInfo.cs" `
        -title "Rhino Mocks Tests $version" `
        -description "Mocking Framework for .NET" `
        -company "Hibernating Rhinos" `
        -product "Rhino Mocks Tests $version" `
        -version $version `
        -clsCompliant "false" `
        -copyright "Hibernating Rhinos & Ayende Rahien 2004 - 2009"
        
    Generate-Assembly-Info `
        -file "$base_dir\Rhino.Mocks.Tests.Model\Properties\AssemblyInfo.cs" `
        -title "Rhino Mocks Tests Model $version" `
        -description "Mocking Framework for .NET" `
        -company "Hibernating Rhinos" `
        -product "Rhino Mocks Tests Model $version" `
        -version $version `
        -clsCompliant "false" `
        -copyright "Hibernating Rhinos & Ayende Rahien 2004 - 2009"
        
    new-item $release_dir -itemType directory 
    new-item $buildartifacts_dir -itemType directory 
    cp $tools_dir\MbUnit\*.* $build_dir
} 

task Compile -depends Init { 
  exec msbuild "/p:OutDir=""$buildartifacts_dir "" $sln_file"
} 

task Test -depends Compile {
  $old = pwd
  cd $build_dir
  exec ".\MbUnit.Cons.exe" "$build_dir\Rhino.Mocks.Tests.dll"
  cd $old        
}

task Merge {
    $old = pwd
    cd $build_dir
    
    Remove-Item Rhino.Mocks.Partial.dll -ErrorAction SilentlyContinue 
    Rename-Item $build_dir\Rhino.Mocks.dll Rhino.Mocks.Partial.dll
    
    & $tools_dir\ILMerge.exe Rhino.Mocks.Partial.dll `
        Castle.DynamicProxy2.dll `
        Castle.Core.dll `
        /out:Rhino.Mocks.dll `
        /t:library `
        "/keyfile:$base_dir\ayende-open-source.snk" `
        "/internalize:$base_dir\ilmerge.exclude"
    if ($lastExitCode -ne 0) {
        throw "Error: Failed to merge assemblies!"
    }
    cd $old
}

task Release -depends Test, Merge {
    & $tools_dir\zip.exe -9 -A -j `
        $release_dir\Rhino.Mocks.zip `
        $build_dir\Rhino.Mocks.dll `
        $build_dir\Rhino.Mocks.xml `
        license.txt `
        acknowledgements.txt
    if ($lastExitCode -ne 0) {
        throw "Error: Failed to execute ZIP command"
    }
}

It is about 50 lines, all told, with a lot of spaces and is quite readable.

This handles the same tasks as the old set of scripts did, and it does this without undue complexity. I like it.

The complexity of unity


This post is about the Rhino Tools project. It has been running for a long time now, over 5 years, and amassed quite a few projects in it.

I really like the codebase in the projects in Rhino Tools, but secondary aspects has been creeping in that made managing the project harder. In particular, putting all the projects in a single repository made it easy, far too easy. Projects had an easy time taking dependencies that they shouldn’t, and the entire build process was… complex, to say the least.

I have been somewhat unhappily tolerant of this so far because while it was annoying, it didn’t actively create problems for me so far. The problems started creeping when I wanted to move Rhino Tools to use NHibernate 2.1. That is when I realized that this is going to be a very painful process, since I have to take on the entire Rhino Tools set of projects in one go, instead of dealing with each of them independently. the fact that so many of the dependencies where in Rhino Commons, to which I have a profound dislike, helped increase my frustration.

There are other things that I find annoying now, Rhino Security is a general purpose library for NHibernate, but it makes a lot of assumptions about how it is going to use, which is wrong. Rhino ETL had a dependency on Rhino Commons because of three classes.

To resolve that, I decided to make a few other changes, taking dependencies is supposed to be a hard process, it is supposed to make you think.

I have been working on splitting the Rhino Tools projects to all its sub projects, so each of them is independent of all the others. That increase the effort of managing all of them as a unit, but decrease the effort of managing them independently.

The current goals are to:

  • Make it simpler to treat each project independently
  • Make it easier to deal with the management of each project (dependencies, build scripts)

There is a side line in which I am also learning to use Git, and there is a high likelihood that the separate Rhino Tools projects will move to github. Suversion’s patching & tracking capabilities annoyed me for the very last time about a week ago.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. RavenDB Sharding (2):
    21 May 2015 - Adding a new shard to an existing cluster, the easy way
  2. The RavenDB Comic Strip (2):
    20 May 2015 - Part II – a team in trouble!
  3. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  4. Interview question (2):
    30 Mar 2015 - fix the index
  5. Excerpts from the RavenDB Performance team report (20):
    20 Feb 2015 - Optimizing Compare – The circle of life (a post-mortem)
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats