Ayende @ Rahien

Refunds available at head office

Hibernating Rhinos #8 - Going Distributed & Building our own Bus

image

Well, I was toying around with the idea for about a month or so, and finally I got around to actually record & editing that.

Highlights:

  • Vastly improved sound quality. I think you'll enjoy it.
  • Vastly extended in time & scope. For some reason, this screencast is longer than many full length movies. We also write our own bus implementation from scratch, and discuss the implementation details there.
  • This is more of a low level discussion, not a high level architectural discussion about why you want a bus (well, I do talk about it a bit, but mostly we implement the bus).
  • The first 45 minutes are dedicated to moving from an old style RPC to an async batching bus approach, that still uses the request / reply. The rest is dedicated to building the one way, message passing, queue based, service bus.
    • There are some interesting challenges there, and I hope you'll make sense of my grunts as I write the code.
    • The last hour or so of the screen cast it live coding, and you get to see how I revert some design decisions as they turn out to be problematic.

The technical details:

  • Total length: An hour and forth minutes(!)
  • Size: 160 MB
  • Code starts on 04:31

Go to download page

Comments

Ray
03/30/2008 01:24 PM by
Ray

Very interesting. Thanks Ayende!

Tuna Toksoz
03/30/2008 02:55 PM by
Tuna Toksoz

The download speed for me is so low(around 4KB), is it normal?

Ayende Rahien
03/30/2008 03:07 PM by
Ayende Rahien

I get 100Kb/s, so I guess it isn't

Ryan Gray
03/30/2008 05:30 PM by
Ryan Gray

I'm trying to view this on XP with Media Player. I have the Camtasia codec installed. All other Hibernating Rhinos episodes work fine on this setup, but this episode is garbled and the colors are wrong/inverted. Suggestions?

Ayende Rahien
03/30/2008 05:40 PM by
Ayende Rahien

This was recorded in Camtasia 5, you might need an updated Codec

Ryan Gray
03/30/2008 06:01 PM by
Ryan Gray

I couldn't find a more updated codec than the one I had. For what it's worth for anyone else who might experience this problem, Winamp has no problem playing this episode. I didn't try any other players.

Now I get to watch it. Thanks Ayende, you're my hero!

Max Bolingbroke
03/30/2008 07:17 PM by
Max Bolingbroke

Thanks for the interesting screencast Ayende!

I did have some problems with the video looking a bit corrupted when watching it using Flip4Mac and Quicktime on OS X. However, I think this is Flip4Macs fault rather than yours.

I have a question: what is your opinion of using multiple simultaneous messages to reduce latency? In particular those three server requests could be issued via a Future and hence you would not have to wait the 1s for a reply in each case. The only cost you would pay is slightly increased network overhead, but the code would be a lot simpler!

Is your use of WeakReferences in the code a bug? Doesn't it mean the garbage collector could collect your RhinoAsyncResults at any time, causing transient failures when you look them up at a later time? Indeed, in any case it would mean that the hashtable will become filled with keys with "empty" weak references. I think a better memory management scheme would involve timing out these RARs after a certain period, though this is of course more complicated. Have I missed something here?

Ayende Rahien
03/30/2008 07:31 PM by
Ayende Rahien

The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.

The WeakReference is not a bug, per se. Yes, there is a memory leak there, but that is easily fixable.

This means that if you are not interest in the result, you can ignore the return value of Process(), and it won't be kept around.

However, if you are interest in it, you will keep a reference to it (usually by calling asynReference.WaitHandle.WaitOne() ). So it wouldn't go away.

Max Bolingbroke
03/30/2008 08:23 PM by
Max Bolingbroke

""

The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.

""

Three messages immediately, yes. I don't deny that it would be more costly in terms of >bandwidth<, but you would not have to suffer the >latency<, which was the main thing you were bothered by. It would also be a lot simpler to implement, since you just reuse the existing code for Futures you blogged about a while ago. I would be tempted to use this approach unless bandwidth had proven to be a problem.

Ayende Rahien
03/30/2008 08:52 PM by
Ayende Rahien

Um, can you tell me what you don't like the batching approach, since that takes care of both latency and bandwidth issues?

It is also very easy to handle futures in this manner, see how we did it for NH

Max Bolingbroke
03/30/2008 09:36 PM by
Max Bolingbroke

Don't get me wrong, I'm not saying that I don't like it! It is fine and doubtless the most efficient solution in terms of latency/bandwidth issues.

All I'm saying is that it imposes a significant overhead in terms of extra code since you seem to need to define those explicit message classes. There are just some situations where development time is more important than having precisely the most efficient transport, and in those cases it would be dead easy to reuse the Futures stuff and a simple synchronous service to deal with the latency, which I agree IS a problem.

Sorry if I didn't make this clear enough earlier. Or do you believe that the extra development time + code to support batching via explicit request/response messages is too small a problem to worry about?

Ayende Rahien
03/30/2008 09:44 PM by
Ayende Rahien

In my experience, remote calls are killer for performance.

Any attempt to hide this will end up biting you in the ass. We have ~20 years of experience with this style of programming, from the early CORBA, DCOM, COM+, WS, etc.

Hiding the network layer makes for piss poor performance.

I actually like the messages classes. Yes, it is more code than the method call, but it is also make it clear how you are communicating, and it opens up some very interesting possibilities.

Edward
04/01/2008 09:54 PM by
Edward

Help me seed this torrent for Hibernating Rhinos 8. These post attract 25-30,000 downloads. Let's use the web more efficiently.

http://www.mininova.org/tor/1286587

Edward

Eli
04/13/2008 12:25 PM by
Eli

When you lock... line number 69 in ClientBus.cs... you mention avoiding lookups in a Dictonary object to avoid lock contention. I'm curious what would be the proper approach assuming that you're worried about locking the entire table rather than the single row that you're interested in?

Ayende Rahien
04/13/2008 12:30 PM by
Ayende Rahien

Building a custom collection type with saner locking semantics

Comments have been closed on this topic.