http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Scooletz, No, this is part of the storage itself, take a look at GetDocumentsAfter implementation http://ayende.com/4765/implementing-background-processes#comment9http://ayende.com/4765/implementing-background-processes#comment9Wed, 09 Feb 2011 09:20:16 GMTAyende, do you use Lucene to query by "last updated timestamp" or is it some feature built in Raven? If so, could you point to the source code where is it held? Do you have any more optimizations in your DB? http://ayende.com/4765/implementing-background-processes#comment8http://ayende.com/4765/implementing-background-processes#comment8Tue, 08 Feb 2011 13:02:44 GMTMatt, It doesn't really solve the problem of how you initiate the process or select the data to process. http://ayende.com/4765/implementing-background-processes#comment7http://ayende.com/4765/implementing-background-processes#comment7Tue, 08 Feb 2011 08:50:05 GMTDo you use or have you considered using the TPL? A combination of a framework for queuing combined with the built-in parallel extensions is a nice fit - sorry if this is a little simplistic an answer; just thought I'd throw it in there. http://ayende.com/4765/implementing-background-processes#comment6http://ayende.com/4765/implementing-background-processes#comment6Tue, 08 Feb 2011 08:47:07 GMTRyan, Yes, there is some danger in that. The issue is a balance between freshness and overall speed. In general, this approach tends to balance itself up pretty well. Let us assume that the update rate in lower than or equal the processing rate. In that case, this pattern gives you results as fresh as possible. That is actually a rare scenario, because it indicate low update rates, and the system would basically sleep part of the time, waiting for updates. If the update rate is greater than the processing rate, it means that the first processing run would pick just the first item, but the second one would pick a full batch, and so on. Another aspect of this system is that you aren't actually requiring any processing power to handle large updates, since we are don't keep any state for each task, just our current position. http://ayende.com/4765/implementing-background-processes#comment5http://ayende.com/4765/implementing-background-processes#comment5Tue, 08 Feb 2011 05:57:40 GMTAlex K, There are many ways that you can handle this scenario. For example, you can use notification triggers in the database, or notify via a pub/sub mechanism. How you do the notification isn't that important, and worst case scenario, you can fall back to polling. Rafal, An answer. http://ayende.com/4765/implementing-background-processes#comment4http://ayende.com/4765/implementing-background-processes#comment4Tue, 08 Feb 2011 05:57:35 GMTSo, is this a question or an answer? http://ayende.com/4765/implementing-background-processes#comment3http://ayende.com/4765/implementing-background-processes#comment3Tue, 08 Feb 2011 05:47:01 GMTIsn't the effect of a 'tight' notification the same of a queue? You are in the risc of doing a single update each time, like with queueing. Also how do you ensure not to miss a notification, when your task is running already? // Ryan http://ayende.com/4765/implementing-background-processes#comment2http://ayende.com/4765/implementing-background-processes#comment2Tue, 08 Feb 2011 02:44:21 GMT"we ensured that this can work by raising a notification whenever the database change" This works when everything is in one place and the DB is closely aware of the background processor. What if the storage the DB is one entity on one machine (that we don't control, as most of us didn't write our own RavenDB we care about this ;) ), and we have the processor running either as a separate process on same machine or on another machine entirely. Can this notify pattern work? What technology would be used to wake a sleeping .Net processor by the DB (or anything else we want to process stuff from) that isn't away of it? http://ayende.com/4765/implementing-background-processes#comment1http://ayende.com/4765/implementing-background-processes#comment1Mon, 07 Feb 2011 22:01:28 GMT