http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Haha, "poor lunch conversation". Thanks for the library, want to go benchmark it in my dev environment and see what kind of stats I can squeze out of it :Phttp://ayende.com/159105/introducing-rhino-events#comment7http://ayende.com/159105/introducing-rhino-events#comment7Mon, 15 Oct 2012 07:27:18 GMT+1 Greg. Last week I also raised concerns and suggested looking at how current implementations of LSM's are built because they make some very important design decisions that aren't implemented here.http://ayende.com/159105/introducing-rhino-events#comment6http://ayende.com/159105/introducing-rhino-events#comment6Fri, 12 Oct 2012 20:21:03 GMTInteresting bits of code. It's fun to play around with it and see how you implemented the different pieces. However, I wonder in which type of scenarios you would consider using it?http://ayende.com/159105/introducing-rhino-events#comment5http://ayende.com/159105/introducing-rhino-events#comment5Fri, 12 Oct 2012 19:27:38 GMTI would not recommend anyone using this code as it stands for anything more than a sample app or something that doesn't have high requirements. There are quite a few issues in it. I have always said an event store is a fun project because you can go anywhere from an afternoon to years on an implementation. I think there is a misunderstanding how people normally use an event stream for event sourcing. They read from it. Then they write to it. They expect optimistic concurrency from another thread having read from then written to the same stream. This is currently not handled. This could be handled as simply as checking the expected previous event but this wouldn't work because the file could be scavenged in between. The way this is generally worked around is a monotonically increasing sequence that gets assigned to an event. This would be relatively trivial to add (just another attribute) The next issue is that I can only read the stream from the beginning to the end or vice versa. If I have a stream with 20m records in it and I have read 14m of them and the power goes out; when I come back up I want to start from 14m (stream.Position = previous; is a Seek() and 14m can be very expensive if you happen to be working with files the OS has not cached for you). This is a hugely expensive operation to redo and the position I could have saved won't help me as the file could get compacted in between. To allow arbitrary access to the stream is a bit more difficult. The naive way would be to use something like a sorteddictionary or dictionary of lists as an index but you will very quickly run out of memory. B+Trees/LSM are quite useful here. Even with the current index (stream name to current position) there is a fairly large problem as it gets large. With 5+m streams you will start seeing large pauses from the serializing out the Dictionary. At around 50m your process will blow up due to 1gb object size limit in CLR Similar to the index issue is that with a dictionary of all keys being stored in memory and taking large numbers of writes per second it is quite likely you will run out of memory if people are using small streams (say I have 10000 sensors and I do a stream every 5 seconds for their data to partition). Performance will also drastically decrease as you use more memory due to GC. A more sinister problem is the scavenge / compaction. It stops the writer. When I have 100mb of events this may be a short pause. When I have 50gb of events this pause may very well turn into minutes. There is also the problem of needing N * N/? disk space in order to do a scavenge (you need both files on disk). With write speeds of 10MB/second it obviously wouldn't take long to make these kinds of huge files especially in a day where we consider a few TB to be small. The general way of handling this is the file gets broken into chunks then each chunk can be scavenged independently (while still allowing reads off it). Chunks can for instance be combined as well as they get smaller (or empty). Another point to bring up is someone wanting to write N events *together* in a transactional fashion to a stream. This sounds like a trivial addition but its less than trivial to implement (especially with some of the other things discussed here). As was mentioned in a previous thread a transaction starts by definition when there is more than one thing to do. There are decades worth of previous art in this space. It might be worth some time looking through it. LSM trees are a good starting point as is some of the older material on various ways of implementing transaction logs. Greghttp://ayende.com/159105/introducing-rhino-events#comment4http://ayende.com/159105/introducing-rhino-events#comment4Fri, 12 Oct 2012 09:52:41 GMTisn't PersistedEventsStorage.WriteToDisk() a spinwait? http://ayende.com/159105/introducing-rhino-events#comment3http://ayende.com/159105/introducing-rhino-events#comment3Thu, 11 Oct 2012 20:19:50 GMT>> "I made for poor lunch conversation, because all I could see in my head was code and IO blocks moving around in interesting ways." Ha! Next time we have dinner, Oren, if I see glazed, wandering eyes, I'll have to find out what new project you're thinking up. :-) Nice work with Rhino Events.http://ayende.com/159105/introducing-rhino-events#comment2http://ayende.com/159105/introducing-rhino-events#comment2Thu, 11 Oct 2012 14:35:29 GMTIn the PersistedEventsStorage ctor, creating a thread via Task.Factory.StartNew is not a very good idea as it will "streal" one from the thread pool, which is subject to constraints and thread injection policies that assume short lived tasks. It would be better to use the LongRunning option when creating the Task or to just create a thread explicitly.http://ayende.com/159105/introducing-rhino-events#comment1http://ayende.com/159105/introducing-rhino-events#comment1Thu, 11 Oct 2012 14:22:31 GMT