The edge case is in the timing
In a previous post, I showed how to get 10x performance boost by batching remote calls. The code included the following method:
This code has a subtle bug in it. Take a look and see if you can find it.
Yes, I’ll wait, honestly.
Go read the code and think about interesting ways to break it.
Okay, found it? Awesome, now let me explain anyway .
This code suffer from the slow arrival syndrome. If we have a new request coming in every 100 ms, then the first request here will languish in the queue for over 25 seconds!
Luckily, the fix is simple:
We just need to make sure that we aren’t waiting for too long. In this case, we will wait a maximum of less than half a second for the messages to go over the wire.
Comments
Damn, and I thought the bug was that,
document
is never declared andval
is never assigned. :) But yes, waiting for 25s before doing any processing would probably be bad.I think you just created a new bug :) sp.ElapsedMilliseconds < 250
Shouldn't it be sp.ElapsedMilliseconds > 250 since that is your exit condition?
Another bug is that if your cancellation token is triggered, then the documents that you removed from the queue are lost.
Hangy, mk & Shaun, Yes, you are correct.
Shaun, If the cancellation token is triggered, the entire operation is cancelled,not just a particular batch
I know this is just a simple example but just in case someone copy and use this code: Maybe new List<RecordMailArgs>(256) before both loops and clearing the list instead of a new one every batch could give you better performance. Not just for the cost of New(); also because Clear does not reset the capacity and prevents autogrouth ( in case you don't use new (Int32) ).
With this implementation and the correct conditions, it's still possible to wait up to 400ms.
Why not compute the remaining time substracting sp.TotalMilliseconds and the initial timeout value ?
I manage to have a stable wait time close to my initial timeout value using different conditions.
Something like
Another approach for so-called smart batching is no-wait on empty queue. If there's nothing left in the queue, proceed with what you get batched and return to polling the queue. This approach is used by EventStore in its storage service. The same can be seen in the LMAX disruptor pattern and Aeron (messaging protocol). The last two actually obtains the last occupied position (depending on the library it's called differently) and proceed with all the items till this point. With this approach you ensure that small batches will be finished in the amortized time of handling a single batch. In your case, you'll always wait for this and additional timeout.
@Oren so basically the good old difference between throttle and debounce?
http://benalman.com/projects/jquery-throttle-debounce-plugin/
(see the diagrams)
Comment preview