Ayende @ Rahien

It's a girl

Handling production errors in a messaging environment

So, today I got the first L2S Prof order. As you can imagine, I was pretty excited about that. However, it turned out that I had actually missed something when I built the backend for handling L2S Prof ordering. The details about what actually went wrong aren’t important (and are embarrassing).

But I logged into the server and checked out what was going on:

image

One of the major design criteria that I had with Rhino Service Bus is that it should be dead easy to handle production. As you can see in the screen shots, I have two messages of interest here, the first one is the actual message, and the second is the error information, which include the full stack trace. Using that, it was a piece of cake to isolate the problem, do the head slapping moment, and deploy a new version out. Once that was done, all I had to do is to move the message back to the processing queue, and I was done.

Just to give you an idea, here is how it looks like on my timeline:

image

I guarantee you that the customer in question didn’t have any idea that there was something wrong with his order processing.

I am loving it.

Comments

Jonathan Vukovich
12/02/2009 11:31 PM by
Jonathan Vukovich

Just had to go through something similar a few weeks ago here at work. It's worth its weight in gold; five times over.

Tyler Burd
12/03/2009 12:03 AM by
Tyler Burd

I LOVE that feature. It has saved me SO many times.

Rafal
12/03/2009 06:01 AM by
Rafal

Oh, come on, you shouldn't be surprised that RSB works... It doesn't look good.

Set
12/03/2009 08:06 AM by
Set

We want to know ( well at least i'd like :P ) the embarrassing part :o

Frans Bouma
12/03/2009 08:50 AM by
Frans Bouma

You might want to blur that SWREG order number in the screenshot.

Ayende Rahien
12/03/2009 09:01 AM by
Ayende Rahien

Frans,

Good point, thanks.

Dave
12/03/2009 09:32 AM by
Dave

I misread the title and thought is was saying ' handling production errors in a messy environment'. Stupid me ;-)

Comments have been closed on this topic.