http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Ryan, As I said, this is the absolute worst possible scenario http://ayende.com/4426/rhino-divan-db-performance#comment10http://ayende.com/4426/rhino-divan-db-performance#comment10Mon, 08 Mar 2010 06:50:32 GMTIsn't a large number of small writes pretty hard to optimise with Lucene, as it means you need to drop and reopen any existing index readers? How are you handling index optimisation? http://ayende.com/4426/rhino-divan-db-performance#comment9http://ayende.com/4426/rhino-divan-db-performance#comment9Sun, 07 Mar 2010 21:43:57 GMTRafal, Indexing time for a single document is practically zero, the cost is actually writing & flushing to the index. A flood of changes doesn't really bother me, you can still access all the data, and the index will keep up with the work being done. The only bad affect is that the index will indicate that it is stale, but there is no real helping that here. 2) Each index is handled separately, more or less, so there would be some indexes that might take a bit longer, but I don't believe that significantly so. 3) Index update is a single threaded operation by nature. http://ayende.com/4426/rhino-divan-db-performance#comment8http://ayende.com/4426/rhino-divan-db-performance#comment8Sun, 07 Mar 2010 19:49:55 GMTRichard, It is WAY too soon to do real benchmarks for Raven. I only did this test to prove that I this approach is valid http://ayende.com/4426/rhino-divan-db-performance#comment7http://ayende.com/4426/rhino-divan-db-performance#comment7Sun, 07 Mar 2010 19:46:32 GMTI understand the delay between changing the document and the time the change is reflected by an index is practically constant. Does it mean that no matter how many documents are updated, the index update delay will be more or less the same? What about a 'flood' of changes at maximum possible rate? 2. What about having several indexes? In many systems I have seen tables having 10-20 indexes for various kinds of search/sorting. What will be the index update delay if there are 20 indexes? 3. Did you measure the insert/update performance in a multithreaded environment? http://ayende.com/4426/rhino-divan-db-performance#comment6http://ayende.com/4426/rhino-divan-db-performance#comment6Sun, 07 Mar 2010 18:53:43 GMTIt might be too soon, but would be cool to see performance benchmarks comparing RDB to NoSQL databases. http://ayende.com/4426/rhino-divan-db-performance#comment5http://ayende.com/4426/rhino-divan-db-performance#comment5Sun, 07 Mar 2010 16:36:07 GMTOops,you are right i forgot that the cost is per a set of modified/added documents. How many indices are you updating with those 5,000 docs? i guess if you have many indices(for many views) the cost per document will grow too,right? http://ayende.com/4426/rhino-divan-db-performance#comment4http://ayende.com/4426/rhino-divan-db-performance#comment4Sun, 07 Mar 2010 15:56:32 GMTUriel, There is a cost per indexing operation, which is pretty high, but that cost isn't affected by the size of the already indexed documents. 5,000 docs or 5 millions, doesn't matter for the indexing op http://ayende.com/4426/rhino-divan-db-performance#comment3http://ayende.com/4426/rhino-divan-db-performance#comment3Sun, 07 Mar 2010 13:11:31 GMTUriel, probably there is a large constant cost, so you cannot multiply it by 1000. http://ayende.com/4426/rhino-divan-db-performance#comment2http://ayende.com/4426/rhino-divan-db-performance#comment2Sun, 07 Mar 2010 11:40:30 GMTThe index update seems high for 5,000 documents. If you scale it to let say 5 million documents,it wil take at least 27 seconds,and this is in the really good case that the indexing scale in a linear way. As you said this is really a small dataset. http://ayende.com/4426/rhino-divan-db-performance#comment1http://ayende.com/4426/rhino-divan-db-performance#comment1Sun, 07 Mar 2010 11:14:08 GMT