Ayende @ Rahien

It's a girl

Implementing Background Processes

I got the following question (originally about RavenDB, but I generalized it a bit):

I'm currently working on a open source project where I need background processing. The main scenarios are:

  • Processing data from a queue of incoming messages, like processing incoming mail that's put in a queue.
  • Processing data from a lot of different web services.

I've worked with scheduling frameworks like quartz.net before to schedule processing but in this case I'm looking at much bigger amounts of processing. It would be nice to add more workers depending on the load like raven db.

I think my main question is what's your experience when building background workers? What should I think about? Is there any framework that can help me?

The first thing to understand is that for data processing, actually implementing queuing is going to be a losing proposition. The absolutely major cost for most data processing task is IO, and the best way to handle that is to handle this via batching. Queues doesn’t really work for this scenario because they make it hard to process a batch of changes in one shot. Queues are natural for “pull from queue, process, move to next message”, which isn’t good when you are processing large amount of information.

The way this is implemented in RavenDB is that I have ensured that there is a cheap way to query by “last updated timestamp”. After that, it means that I am able to issues queries such as:

Give me the next batch of updated documents since update point 121.

Those queries are very cheap (they are fully indexed queries at the storage level).

Following that, each data processing task merely need to keep track of the last update point that it processed. Things get a little more complex when you assume that there can be periods of time where no activity happens, since you want to avoid polling in that scenario.

With RavenDB, if a processing task doesn’t find anything to process, it goes to sleep, and we ensured that this can work by raising a notification whenever the database change, in which case we can wake the waiting tasks. This approach allows us to efficiently process data without waiting for scheduled tasks (which result in update delays), without polling (which consume additional resources) and without complex logic (scheduling, determining what changed, queues, etc).

I find this to be quite an elegant solution.

Comments

Alex K
02/07/2011 10:01 PM by
Alex K

"we ensured that this can work by raising a notification whenever the database change"

This works when everything is in one place and the DB is closely aware of the background processor. What if the storage the DB is one entity on one machine (that we don't control, as most of us didn't write our own RavenDB we care about this ;) ), and we have the processor running either as a separate process on same machine or on another machine entirely. Can this notify pattern work? What technology would be used to wake a sleeping .Net processor by the DB (or anything else we want to process stuff from) that isn't away of it?

Ryan Heath
02/08/2011 02:44 AM by
Ryan Heath

Isn't the effect of a 'tight' notification the same of a queue?

You are in the risc of doing a single update each time, like with queueing.

Also how do you ensure not to miss a notification, when your task is running already?

// Ryan

Rafal
02/08/2011 05:47 AM by
Rafal

So, is this a question or an answer?

Ayende Rahien
02/08/2011 05:57 AM by
Ayende Rahien

Alex K,

There are many ways that you can handle this scenario.

For example, you can use notification triggers in the database, or notify via a pub/sub mechanism.

How you do the notification isn't that important, and worst case scenario, you can fall back to polling.

Rafal,

An answer.

Ayende Rahien
02/08/2011 05:57 AM by
Ayende Rahien

Ryan,

Yes, there is some danger in that. The issue is a balance between freshness and overall speed.

In general, this approach tends to balance itself up pretty well.

Let us assume that the update rate in lower than or equal the processing rate. In that case, this pattern gives you results as fresh as possible.

That is actually a rare scenario, because it indicate low update rates, and the system would basically sleep part of the time, waiting for updates.

If the update rate is greater than the processing rate, it means that the first processing run would pick just the first item, but the second one would pick a full batch, and so on.

Another aspect of this system is that you aren't actually requiring any processing power to handle large updates, since we are don't keep any state for each task, just our current position.

Matt Salmon
02/08/2011 08:47 AM by
Matt Salmon

Do you use or have you considered using the TPL? A combination of a framework for queuing combined with the built-in parallel extensions is a nice fit - sorry if this is a little simplistic an answer; just thought I'd throw it in there.

Ayende Rahien
02/08/2011 08:50 AM by
Ayende Rahien

Matt,

It doesn't really solve the problem of how you initiate the process or select the data to process.

Scooletz
02/08/2011 01:02 PM by
Scooletz

Ayende, do you use Lucene to query by "last updated timestamp" or is it some feature built in Raven? If so, could you point to the source code where is it held? Do you have any more optimizations in your DB?

Ayende Rahien
02/09/2011 09:20 AM by
Ayende Rahien

Scooletz,

No, this is part of the storage itself, take a look at GetDocumentsAfter implementation

Comments have been closed on this topic.