http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660+1 Clemenshttp://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment26http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment26Wed, 10 Oct 2012 06:20:00 GMTKelly, Let us say that you want to read an existing event stream. It has 10,000,000 events in it. And it takes an hour to iterate over it. During that hour, you have what is effectively snapshot isolation over this. You can only see if as it was when you started the read. This is because of the way the system works. You get the latest file offset from the in memory data structure. Then you start moving backward. The file is immutable, and you are always moving backward, there is no chance of you seeing other data.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment25http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment25Tue, 09 Oct 2012 17:12:04 GMTKelly, Isolation here works by having the reads always work on top of the idToPos dictionary, which contains the file positions. Only after we flush to disk will we update that dictionary. Therefor, reads see the "old" state, and only after the disk flush "tx commit", is that state visible.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment24http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment24Tue, 09 Oct 2012 17:09:54 GMTIf isolation guarantees were provided, if I were to start a read operation where I want to iterate over the stream of 100 million events to make some sort of calculation that may take an hour there would be no possible way for me to retrieve events committed via other transactions during the same time period. Isolation by a cursor position isn't good enough either because you could guess what the next sequence is and get an event that didn't exist at the time the reads began.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment23http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment23Tue, 09 Oct 2012 16:10:51 GMTI'll observe that the use of ACID here seems to be pure marketing. Since there is no batching, this is obviously about single records that are written into a fairly simple store. That's basic consistency. The common notion of transactions starts at two or more things that need to happen at the same time.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment22http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment22Tue, 09 Oct 2012 15:51:01 GMTAyende, That statement about isolation doesn't make sense to me. Perhaps you can elaborate? If isolation is guaranteed, how are you isolating reads from writes? http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment21http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment21Tue, 09 Oct 2012 15:22:55 GMTto be clear what you are describing here has no form of dependency between any two things being written or read. What I read has no dependency on what I write When I write multiple things they can have no dependency on each other Well yeah you can make this very fast. Its just a log file. You can get it even faster very simply. Put it on 5 drives.No write or read from any drive can possibly have a dependency on a read/write from another drive http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment20http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment20Tue, 09 Oct 2012 15:18:00 GMTAtomicity - We don't support the notion of a batch, so either we have successfully written an even to the disk, or we didn't. There isn't a middle ground. Most of the time you have to support a batch. Consistency - We don't make the newly written event visible to the app code until we have saved that to disk, therefor, we are consistent. This is an interesting version of consistency. There are basically no rules. Add a small rule. I can put an expected version (optimistic concurrency) for the stream when writing and things get quite a bit trickier especially with supporting batcheshttp://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment19http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment19Tue, 09 Oct 2012 15:12:54 GMTKelly & Greg, Atomicity - We don't support the notion of a batch, so either we have successfully written an even to the disk, or we didn't. There isn't a middle ground. Consistency - We don't make the newly written event visible to the app code until we have saved that to disk, therefor, we are consistent. Isolated - Same as before, an event can't touch another event, or impact it in any way. Durable - Data goes to disk, and you can verify when it is actually flushed there.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment18http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment18Tue, 09 Oct 2012 14:46:01 GMTAlso from my understanding it doesn't have Atomicity from ACID either since a batch doesn't fail as a unit. As far as I can tell you can get partial writes.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment17http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment17Tue, 09 Oct 2012 13:26:30 GMTGreg, +1 about consistency. From my understanding I didn't see any isolation from ACID implementation either. If I'm incorrect please point it out to me as I'd like to learn how it's implemented but as far as I could tell I couldn't see any.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment16http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment16Tue, 09 Oct 2012 13:15:40 GMTTo be clear on my consistency comment: "The consistency property ensures that any transaction will bring the database from one valid state to another. Any data written to the database must be valid according to all defined rules, including but not limited to constraints, cascades, triggers, and any combination thereof." I guess if you have no rules at all to make consistent then you are consistent but most systems have some rules to actually make consistent :) http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment15http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment15Tue, 09 Oct 2012 08:43:41 GMTAyende, This is how most systems are built under the covers. Check out cass, bitcask or even Essent that you use in RavenDb. Bitcask is OSS you can see it does essentially the code you have written (well with more concerns but same general idea). The event store works this way as well and is OSS (BSD license) you could even use core bits in ravendb. "The entire thing work with ACID guarantees" There is no C here. If by ACID you mean Durable then yes I agree durability has been reached. What do you intend to make C? For us our indexes are C. As you know from Raven indexes that are C can be expensive to get. Even doing something as simple as keeping a current version number of say the document you are writing to to provide basic optimistic concurrency is actually reasonably difficult and slow to provide on top when we start talking about not being able to fit all your keys in memory. Especially since in order to maintain your consistency it would either run on the same thread as your write or you would end up with lots of very intricate locking code. http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment14http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment14Tue, 09 Oct 2012 08:26:11 GMTStill disagree that weekend hacks ALWAYS outperform established software. I don't buy into context-insensitive absolutes. Lazy thinking.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment13http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment13Tue, 09 Oct 2012 05:55:35 GMTNick, No, that is actually pretty good. It is _easy_ to get good perf when you don't have to consider all of the other stuff that you want done.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment12http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment12Mon, 08 Oct 2012 22:12:06 GMTRoy, The entire thing work with ACID guarantees. And the latency is max 200 ms under very heavy load with read latency that is _very_ low.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment11http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment11Mon, 08 Oct 2012 22:11:14 GMT@Roy, Ayende blog about those cases and more in a few posts last week about how certain features in RavenDb were implemented.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment10http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment10Mon, 08 Oct 2012 18:12:44 GMT"That's why weekend hacks always out perform well established software" Might want to rephrase that just a touch.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment9http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment9Mon, 08 Oct 2012 18:10:07 GMTI agree with Roy. People make careers out of writing production quality transactional storage engines. The benchmark numbers aren't interesting to me on a storage backend hacked in a weekend. The compaction strategy and failure handling concerns me. I don't think it would do well in a busy system over time. All of these intricate details affect benchmarks quite a bit. It's easy to get favourable numbers by taking liberties of things required by production grade software that can be ignored in a benchmark. That's why weekend hacks always out perform well established software.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment8http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment8Mon, 08 Oct 2012 16:32:34 GMTNot to be snarky, but 'just getting throughput' is the easy part, usually. Especially with the excellent language support you get in .NET. Getting everything to work with low latencies, transactionally and without corrupting when the power goes out is decidedly non-trivial. Or is this just an exercise in what the upper bounds of the performance can be?http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment7http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment7Mon, 08 Oct 2012 13:45:18 GMTNow you're thinking like me, which is dangerous - you will eventually end up writing your storage engine for RavenDB or something, heh ;-)http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment6http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment6Mon, 08 Oct 2012 13:25:12 GMTThe title should read "Get thee..." or "Get thyself...". Just sayin'http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment5http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment5Mon, 08 Oct 2012 12:33:10 GMTThe write model should also work rather nicely with old magnetic-tape storage. If we use the numbers provided by wikipedia on the UNISERVO (128 chars / inch density), then in order to sustain the write speed you observed, based on a message size of 50 chars, you would need 456.4 metres of tape per second. Hence the tape would have to move at a speed of roughly Mach 1.34. Assuming those tapes were available in length of 730 metres, you would have to change the tape every 1.6 seconds.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment4http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment4Mon, 08 Oct 2012 12:26:05 GMTIt the code in question publicly available? Would be interesting to see it in context.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment3http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment3Mon, 08 Oct 2012 12:18:26 GMTI should add another beauty of the model is if you DONT want to flush. You can just allow read aheads.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment2http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment2Wed, 03 Oct 2012 05:44:51 GMTAyende, Yes the mechanism is age old and works very well (lots of systems use similar mechanisms sql server, bitcask in riak, cassandra), . A few comments on your implementation. You are flushing once per second or when the queue is empty in the code I have. This second one is a great optimization for lowering request latency when you don't have tons of requests. if(hadWrites) { if ((DateTime.UtcNow - lastWrite).TotalSeconds > 1) { While this will give you very high throughput it also introduces a massive amount of latency under the load you gave it (assuming a dual append commit model you are talking 1-2 seconds of latency per transaction assuming durability). If you look in the Event Store code there is actually a heuristic for working around this by looking at the queue and how long your disk takes to fsync. Another comment worth mentioning is that you use streamSource.Flush(file); From looking at the code I was sent by you flush is implemented by: public void Flush(Stream stream) { //((FileStream) stream).Flush(true); } This means that your test is not durable and is actually likely running solely in memory albeit kernel level OS memory but memory. Putting on the flush will knock down your performance quite a bit. The other thing you have to be very careful about here is that very often consumer hardware will deliberately lie when you tell it to flush and will "say sure I did it" even though it didn't. :) In general though yes reading and writing sequential streams is very very fast.http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment1http://ayende.com/159009/get-thou-out-of-my-head-damn-idea#comment1Wed, 03 Oct 2012 05:42:05 GMT