Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 18 | Comments: 72

filter by tags archive

Hibernating Rhinos #8 - Going Distributed & Building our own Bus

time to read 2 min | 271 words

image

Well, I was toying around with the idea for about a month or so, and finally I got around to actually record & editing that.

Highlights:

  • Vastly improved sound quality. I think you'll enjoy it.
  • Vastly extended in time & scope. For some reason, this screencast is longer than many full length movies. We also write our own bus implementation from scratch, and discuss the implementation details there.
  • This is more of a low level discussion, not a high level architectural discussion about why you want a bus (well, I do talk about it a bit, but mostly we implement the bus).
  • The first 45 minutes are dedicated to moving from an old style RPC to an async batching bus approach, that still uses the request / reply. The rest is dedicated to building the one way, message passing, queue based, service bus.
    • There are some interesting challenges there, and I hope you'll make sense of my grunts as I write the code.
    • The last hour or so of the screen cast it live coding, and you get to see how I revert some design decisions as they turn out to be problematic.

The technical details:

  • Total length: An hour and forth minutes(!)
  • Size: 160 MB
  • Code starts on 04:31

Go to download page


Comments

Ray
Ray

Very interesting. Thanks Ayende!

Tuna Toksoz

The download speed for me is so low(around 4KB), is it normal?

Ayende Rahien

I get 100Kb/s, so I guess it isn't

Ryan Gray

I'm trying to view this on XP with Media Player. I have the Camtasia codec installed. All other Hibernating Rhinos episodes work fine on this setup, but this episode is garbled and the colors are wrong/inverted. Suggestions?

Ayende Rahien

This was recorded in Camtasia 5, you might need an updated Codec

Ryan Gray

I couldn't find a more updated codec than the one I had. For what it's worth for anyone else who might experience this problem, Winamp has no problem playing this episode. I didn't try any other players.

Now I get to watch it. Thanks Ayende, you're my hero!

Max Bolingbroke

Thanks for the interesting screencast Ayende!

I did have some problems with the video looking a bit corrupted when watching it using Flip4Mac and Quicktime on OS X. However, I think this is Flip4Macs fault rather than yours.

I have a question: what is your opinion of using multiple simultaneous messages to reduce latency? In particular those three server requests could be issued via a Future and hence you would not have to wait the 1s for a reply in each case. The only cost you would pay is slightly increased network overhead, but the code would be a lot simpler!

Is your use of WeakReferences in the code a bug? Doesn't it mean the garbage collector could collect your RhinoAsyncResults at any time, causing transient failures when you look them up at a later time? Indeed, in any case it would mean that the hashtable will become filled with keys with "empty" weak references. I think a better memory management scheme would involve timing out these RARs after a certain period, though this is of course more complicated. Have I missed something here?

Ayende Rahien

The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.

The WeakReference is not a bug, per se. Yes, there is a memory leak there, but that is easily fixable.

This means that if you are not interest in the result, you can ignore the return value of Process(), and it won't be kept around.

However, if you are interest in it, you will keep a reference to it (usually by calling asynReference.WaitHandle.WaitOne() ). So it wouldn't go away.

Max Bolingbroke

""

The idea was to save network traffic, so we introduced IBatchingBus, then we made it async.

I am not sure what you mean by simultaneous messages, sending three separate messages immediately ? That would still be more costly than batching them, for both client and server.

""

Three messages immediately, yes. I don't deny that it would be more costly in terms of >bandwidth<, but you would not have to suffer the >latency<, which was the main thing you were bothered by. It would also be a lot simpler to implement, since you just reuse the existing code for Futures you blogged about a while ago. I would be tempted to use this approach unless bandwidth had proven to be a problem.

Ayende Rahien

Um, can you tell me what you don't like the batching approach, since that takes care of both latency and bandwidth issues?

It is also very easy to handle futures in this manner, see how we did it for NH

Max Bolingbroke

Don't get me wrong, I'm not saying that I don't like it! It is fine and doubtless the most efficient solution in terms of latency/bandwidth issues.

All I'm saying is that it imposes a significant overhead in terms of extra code since you seem to need to define those explicit message classes. There are just some situations where development time is more important than having precisely the most efficient transport, and in those cases it would be dead easy to reuse the Futures stuff and a simple synchronous service to deal with the latency, which I agree IS a problem.

Sorry if I didn't make this clear enough earlier. Or do you believe that the extra development time + code to support batching via explicit request/response messages is too small a problem to worry about?

Ayende Rahien

In my experience, remote calls are killer for performance.

Any attempt to hide this will end up biting you in the ass. We have ~20 years of experience with this style of programming, from the early CORBA, DCOM, COM+, WS, etc.

Hiding the network layer makes for piss poor performance.

I actually like the messages classes. Yes, it is more code than the method call, but it is also make it clear how you are communicating, and it opens up some very interesting possibilities.

Edward

Help me seed this torrent for Hibernating Rhinos 8. These post attract 25-30,000 downloads. Let's use the web more efficiently.

http://www.mininova.org/tor/1286587

Edward

Eli
Eli

When you lock... line number 69 in ClientBus.cs... you mention avoiding lookups in a Dictonary object to avoid lock contention. I'm curious what would be the proper approach assuming that you're worried about locking the entire table rather than the single row that you're interested in?

Ayende Rahien

Building a custom collection type with saner locking semantics

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. RavenDB 3.0 New Stable Release - 17 hours from now
  2. Production postmortem: The industry at large - about one day from now
  3. The insidious cost of allocations - 3 days from now
  4. Buffer allocation strategies: A possible solution - 6 days from now
  5. Buffer allocation strategies: Explaining the solution - 7 days from now

And 3 more posts are pending...

There are posts all the way to Sep 11, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    01 Sep 2015 - The case of the lying configuration file
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats