Ayende @ Rahien

It's a girl

A messaging problem: Order in the bus

In NH Prof, I have structured the application around the idea of message passing. I am not yet in the erlang world (which requires a framework that would keep hold of the state), but I am nearer than before.

The back end of the profiler is listening to the event streams generated by NHibernate. We get things like:

  • Session opened on thread #1
  • SQL Executed on thread #1
    • SELECT * FROM Customers
  • Session closed on thread #1

I am taking that and turning it into something that is more easily understandable.

Now, as you can imagine, order is pretty important here.

Currently I am dealing with this by assuming order in the stream, and ensuring that the bus will dispatch messages in order (and that recursive message dispatch is handled in place). This works, but I don't think that I like it much. Especially in light of the previous problem that I outlined.

Another option would be to avoid the order assumption, and use the timestamp in order to reorder the stream. That might be a challenge to solve, though.



10/22/2008 07:15 PM by

But where's the problem? You are monitoring a sequential stream of events so the order is known and never changes. It gets mixed up because of the data transmission method you chose (async loggers). Fix the data transmission and the problem will disappear.

Neal Blomfield
10/22/2008 07:28 PM by
Neal Blomfield

The nastiest issue I can see with relying on the timestamp is, at what point do you "know" you have all of the messages between two points in time such that you can begin to order them?

Or do you deal with this at the last possible moment such that each message is processed and your visualisation of the process is no longer a queue but a list where any message may be inserted at any point (according to the timestamp) therefore forcing an update of any dependant metrics?

Why is order important? What aspect of the application requires that messages must be processed in-order? Could you not argue that the real-time nature of the profiling is such that it is more likely capture and review within a small timespan (10 - 15s) and that some out of order processing (so long as the order is eventually corrected) will not cause a significant distortion of the perception provided by the profiling?

Interesting problem.

Brian Hartsock
10/22/2008 09:42 PM by
Brian Hartsock

In my experience with async messaging, although more with messaging systems like ActiveMQ, assuming order can be a bad assumption. It shouldn't be a problem that can't be overcome though.

In my case, a simple integration test that spit out a few hundred thousand messages and checked for order was enough to know whether or not it is a good assumption. If this doesn't suit you, then I would lean on the side of figuring out a solution that doesn't assume order.

Bad assumptions suck in the long run. Much worse than most bugs in my opinion.

Ayende Rahien
10/23/2008 07:12 PM by
Ayende Rahien


You really want to be able to see the statements for the profiler in the same order that the app executed them, and you really want to be able to associate the statements to the proper session.

I solved the issue by relying on the timestamp, and ensuring that I can handle out of order (to some level) messages.

Neal Blomfield
10/23/2008 07:27 PM by
Neal Blomfield

Collection and processing != display.

You could process the messages as they arrived within a context and reorder them according to the timestamp, updating the context as required. However associating messages with an appropriate session is not a constraint I had forseen, now your choice makes a lot more sense

10/24/2008 01:42 AM by

Just realize that the timestamp-based solution only works if the event stream is coming from a single server (i.e. relying on the accuracy of clocks across multiple machines is not a good solution)..

If you plan on ordering streams from multiple servers, you'd need to look at something like vectors instead of timestamps..

One other thing to realize, timestamps give you ordering. Vectors give you absolute ordering (or causality for other non-sql circumstances). With the vectors, you know when receiving B that A directly preceded it. That solves other crazy scenarios like, "do i have all the messages that preceeded the one i'm about to display"

Although you may not care. And it may not be relevant to NH Prof :-)

Ayende Rahien
10/24/2008 01:47 AM by
Ayende Rahien

I am listening to an event stream generated by a single app instance on a single machine. I can rely on timestamps for orders, yes

10/24/2008 06:48 PM by

id use stamping, but not with time. use a logical clock instead (stamp with int++)

that way ordering is supported by thee messsag logically, instead of assuming syynched cpus or just that ordering is more thann x wrong messagges in a timespan y.

sorry for typos, answering on aa pdaa.

Comments have been closed on this topic.