Ayende @ Rahien

Hi!
My name is Ayende Rahien
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

@

Posts: 5,947 | Comments: 44,541

filter by tags archive

Handling production errors in a messaging environment


So, today I got the first L2S Prof order. As you can imagine, I was pretty excited about that. However, it turned out that I had actually missed something when I built the backend for handling L2S Prof ordering. The details about what actually went wrong aren’t important (and are embarrassing).

But I logged into the server and checked out what was going on:

image

One of the major design criteria that I had with Rhino Service Bus is that it should be dead easy to handle production. As you can see in the screen shots, I have two messages of interest here, the first one is the actual message, and the second is the error information, which include the full stack trace. Using that, it was a piece of cake to isolate the problem, do the head slapping moment, and deploy a new version out. Once that was done, all I had to do is to move the message back to the processing queue, and I was done.

Just to give you an idea, here is how it looks like on my timeline:

image

I guarantee you that the customer in question didn’t have any idea that there was something wrong with his order processing.

I am loving it.


Comments

Jonathan Vukovich

Just had to go through something similar a few weeks ago here at work. It's worth its weight in gold; five times over.

Tyler Burd

I LOVE that feature. It has saved me SO many times.

Rafal

Oh, come on, you shouldn't be surprised that RSB works... It doesn't look good.

Set
Set

We want to know ( well at least i'd like :P ) the embarrassing part :o

Frans Bouma

You might want to blur that SWREG order number in the screenshot.

Dave

I misread the title and thought is was saying ' handling production errors in a messy environment'. Stupid me ;-)

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. RavenDB Sharding (3):
    22 May 2015 - Adding a new shard to an existing cluster, splitting the shard
  2. The RavenDB Comic Strip (2):
    20 May 2015 - Part II – a team in trouble!
  3. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  4. Interview question (2):
    30 Mar 2015 - fix the index
  5. Excerpts from the RavenDB Performance team report (20):
    20 Feb 2015 - Optimizing Compare – The circle of life (a post-mortem)
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats