Ayende @ Rahien

Hi!
My name is Ayende Rahien
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,949 | Comments: 44,548

filter by tags archive

A messaging problem: Order in the bus


In NH Prof, I have structured the application around the idea of message passing. I am not yet in the erlang world (which requires a framework that would keep hold of the state), but I am nearer than before.

The back end of the profiler is listening to the event streams generated by NHibernate. We get things like:

  • Session opened on thread #1
  • SQL Executed on thread #1
    • SELECT * FROM Customers
  • Session closed on thread #1

I am taking that and turning it into something that is more easily understandable.

Now, as you can imagine, order is pretty important here.

Currently I am dealing with this by assuming order in the stream, and ensuring that the bus will dispatch messages in order (and that recursive message dispatch is handled in place). This works, but I don't think that I like it much. Especially in light of the previous problem that I outlined.

Another option would be to avoid the order assumption, and use the timestamp in order to reorder the stream. That might be a challenge to solve, though.

Thoughts?


Comments

nw
nw

But where's the problem? You are monitoring a sequential stream of events so the order is known and never changes. It gets mixed up because of the data transmission method you chose (async loggers). Fix the data transmission and the problem will disappear.

Neal Blomfield

The nastiest issue I can see with relying on the timestamp is, at what point do you "know" you have all of the messages between two points in time such that you can begin to order them?

Or do you deal with this at the last possible moment such that each message is processed and your visualisation of the process is no longer a queue but a list where any message may be inserted at any point (according to the timestamp) therefore forcing an update of any dependant metrics?

Why is order important? What aspect of the application requires that messages must be processed in-order? Could you not argue that the real-time nature of the profiling is such that it is more likely capture and review within a small timespan (10 - 15s) and that some out of order processing (so long as the order is eventually corrected) will not cause a significant distortion of the perception provided by the profiling?

Interesting problem.

Brian Hartsock

In my experience with async messaging, although more with messaging systems like ActiveMQ, assuming order can be a bad assumption. It shouldn't be a problem that can't be overcome though.

In my case, a simple integration test that spit out a few hundred thousand messages and checked for order was enough to know whether or not it is a good assumption. If this doesn't suit you, then I would lean on the side of figuring out a solution that doesn't assume order.

Bad assumptions suck in the long run. Much worse than most bugs in my opinion.

Ayende Rahien

Neal,

You really want to be able to see the statements for the profiler in the same order that the app executed them, and you really want to be able to associate the statements to the proper session.

I solved the issue by relying on the timestamp, and ensuring that I can handle out of order (to some level) messages.

Neal Blomfield

Collection and processing != display.

You could process the messages as they arrived within a context and reorder them according to the timestamp, updating the context as required. However associating messages with an appropriate session is not a constraint I had forseen, now your choice makes a lot more sense

Evan

Just realize that the timestamp-based solution only works if the event stream is coming from a single server (i.e. relying on the accuracy of clocks across multiple machines is not a good solution)..

If you plan on ordering streams from multiple servers, you'd need to look at something like vectors instead of timestamps..

One other thing to realize, timestamps give you ordering. Vectors give you absolute ordering (or causality for other non-sql circumstances). With the vectors, you know when receiving B that A directly preceded it. That solves other crazy scenarios like, "do i have all the messages that preceeded the one i'm about to display"

Although you may not care. And it may not be relevant to NH Prof :-)

Ayende Rahien

I am listening to an event stream generated by a single app instance on a single machine. I can rely on timestamps for orders, yes

morten

id use stamping, but not with time. use a logical clock instead (stamp with int++)

that way ordering is supported by thee messsag logically, instead of assuming syynched cpus or just that ordering is more thann x wrong messagges in a timespan y.

sorry for typos, answering on aa pdaa.

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
  2. Special Offer (2):
    27 May 2015 - 29% discount for all our products
  3. RavenDB Sharding (3):
    22 May 2015 - Adding a new shard to an existing cluster, splitting the shard
  4. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  5. Interview question (2):
    30 Mar 2015 - fix the index
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats