Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,026 | Comments: 44,842

filter by tags archive

Hibernating Rhinos #8 - Going Distributed & Building our own Bus

time to read 2 min | 271 words


Well, I was toying around with the idea for about a month or so, and finally I got around to actually record & editing that.


  • Vastly improved sound quality. I think you'll enjoy it.
  • Vastly extended in time & scope. For some reason, this screencast is longer than many full length movies. We also write our own bus implementation from scratch, and discuss the implementation details there.
  • This is more of a low level discussion, not a high level architectural discussion about why you want a bus (well, I do talk about it a bit, but mostly we implement the bus).
  • The first 45 minutes are dedicated to moving from an old style RPC to an async batching bus approach, that still uses the request / reply. The rest is dedicated to building the one way, message passing, queue based, service bus.
    • There are some interesting challenges there, and I hope you'll make sense of my grunts as I write the code.
    • The last hour or so of the screen cast it live coding, and you get to see how I revert some design decisions as they turn out to be problematic.

The technical details:

  • Total length: An hour and forth minutes(!)
  • Size: 160 MB
  • Code starts on 04:31

Go to download page



Very interesting. Thanks Ayende!

Tuna Toksoz

The download speed for me is so low(around 4KB), is it normal?

Ayende Rahien

I get 100Kb/s, so I guess it isn't

Ryan Gray

I'm trying to view this on XP with Media Player. I have the Camtasia codec installed. All other Hibernating Rhinos episodes work fine on this setup, but this episode is garbled and the colors are wrong/inverted. Suggestions?

Ayende Rahien

This was recorded in Camtasia 5, you might need an updated Codec

Ryan Gray

I couldn't find a more updated codec than the one I had. For what it's worth for anyone else who might experience this problem, Winamp has no problem playing this episode. I didn't try any other players.

Now I get to watch it. Thanks Ayende, you're my hero!

Max Bolingbroke

Thanks for the interesting screencast Ayende!

I did have some problems with the video looking a bit corrupted when watching it using Flip4Mac and Quicktime on OS X. However, I think this is Flip4Macs fault rather than yours.

I have a question: what is your opinion of using multiple simultaneous messages to reduce latency? In particular those three server requests could be issued via a Future and hence you would not have to wait the 1s for a reply in each case. The only cost you would pay is slightly increased network overhead, but the code would be a lot simpler!

Is your use of WeakReferences in the code a bug? Doesn't it mean the garbage collector could collect your RhinoAsyncResults at any time, causing transient failures when you look them up at a later time? Indeed, in any case it would mean that the hashtable will become filled with keys with "empty" weak references. I think a better memory management scheme would involve timing out these RARs after a certain period, though this is of course more complicated. Have I missed something here?

Ayende Rahien

The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.

The WeakReference is not a bug, per se. Yes, there is a memory leak there, but that is easily fixable.

This means that if you are not interest in the result, you can ignore the return value of Process(), and it won't be kept around.

However, if you are interest in it, you will keep a reference to it (usually by calling asynReference.WaitHandle.WaitOne() ). So it wouldn't go away.

Max Bolingbroke


The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.


Three messages immediately, yes. I don't deny that it would be more costly in terms of >bandwidth<, but you would not have to suffer the >latency<, which was the main thing you were bothered by. It would also be a lot simpler to implement, since you just reuse the existing code for Futures you blogged about a while ago. I would be tempted to use this approach unless bandwidth had proven to be a problem.

Ayende Rahien

Um, can you tell me what you don't like the batching approach, since that takes care of both latency and bandwidth issues?

It is also very easy to handle futures in this manner, see how we did it for NH

Max Bolingbroke

Don't get me wrong, I'm not saying that I don't like it! It is fine and doubtless the most efficient solution in terms of latency/bandwidth issues.

All I'm saying is that it imposes a significant overhead in terms of extra code since you seem to need to define those explicit message classes. There are just some situations where development time is more important than having precisely the most efficient transport, and in those cases it would be dead easy to reuse the Futures stuff and a simple synchronous service to deal with the latency, which I agree IS a problem.

Sorry if I didn't make this clear enough earlier. Or do you believe that the extra development time + code to support batching via explicit request/response messages is too small a problem to worry about?

Ayende Rahien

In my experience, remote calls are killer for performance.

Any attempt to hide this will end up biting you in the ass. We have ~20 years of experience with this style of programming, from the early CORBA, DCOM, COM+, WS, etc.

Hiding the network layer makes for piss poor performance.

I actually like the messages classes. Yes, it is more code than the method call, but it is also make it clear how you are communicating, and it opens up some very interesting possibilities.


Help me seed this torrent for Hibernating Rhinos 8. These post attract 25-30,000 downloads. Let's use the web more efficiently.




When you lock... line number 69 in ClientBus.cs... you mention avoiding lookups in a Dictonary object to avoid lock contention. I'm curious what would be the proper approach assuming that you're worried about locking the entire table rather than the single row that you're interested in?

Ayende Rahien

Building a custom collection type with saner locking semantics

Comment preview

Comments have been closed on this topic.


No future posts left, oh my!


  1. Technical observations from my wife (3):
    13 Nov 2015 - Production issues
  2. Production postmortem (13):
    13 Nov 2015 - The case of the “it is slow on that machine (only)”
  3. Speaking (5):
    09 Nov 2015 - Community talk in Kiev, Ukraine–What does it take to be a good developer
  4. Find the bug (5):
    11 Sep 2015 - The concurrent memory buster
  5. Buffer allocation strategies (3):
    09 Sep 2015 - Bad usage patterns
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats