Ayende @ Rahien

Hi!
My name is Ayende Rahien
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,949 | Comments: 44,548

filter by tags archive

The RavenDB indexing processOptimization–De-parallelizing work


One of the major dangers in doing perf work is that you have a scenario, and you optimize the hell out of that scenario. It is actually pretty easy to do without even noticing it. The problem is that when you do things like that, you are likely to be optimizing a single scenario to perform really well, but you are hurting the overall system performance.

In this example, we have moved heaven and earth to make sure that we are indexing things as fast as possible, and we tested with 3 indexes, on an 4 cores machine. As it turned out, we actually had improved things, for that particular scenario.

Using the same test case on a single core machine was suddenly far more heavy weight, because we were pushing a lot of work at the same time. More than the machine could process. The end result was that it actually got there, but much more slowly than if we would have run things sequentially.

Of course, I give you the outliers, but those are good indicators for what we found out. Initially, we thought that we could resolve that by using the TPL’s MaxDegreeOfParallelism, but it turned out to be more complex than that. We have IO bound and we have CPU bound tasks that we need to execute, and trying to execute IO heavy tasks with this would actually cause issues in this scenario.

We had to manually throttle things ourselves, both to ensure limited number of parallel work, and because we have a lot more information about the actual tasks than the TPL have. We can schedule them in a way that is far more efficient because we can tell what is actually going on.

The end result is that we are actually using less parallelism, overall, but in a more efficient manner.

In my next post, I’ll discuss the auto batch tuning support, which allows us to do some really amazing things from the point of view of system performance.

More posts in "The RavenDB indexing process" series:

  1. (24 Apr 2012) Optimization–Tuning? Why, we have auto tuning
  2. (23 Apr 2012) Optimization–Getting documents from disk
  3. (20 Apr 2012) Optimization–De-parallelizing work
  4. (19 Apr 2012) Optimization–Parallelizing work
  5. (18 Apr 2012) Optimization

Comments

gandjustas

What actual API you used for parallel computations? Parallel.ForEach? It's not suitable for IO-bound concurrency. For IO-bound you should use Tasks.

Daniel O

Enjoying these posts on the ongoing development of RavenDB. Could you imagine if the SQL Server or Oracle devs did posts like this?

Gene Hughson

Performance tuning on a line of business app is expensive and labor-intensive to get right: you really need a comprehensive suite of load tests on the same hardware profile as production using an equivalent network profile - easy-peasy. I can only imagine the headache involved with a more general-purpose tool like Raven. Like Daniel above, I'm enjoying the peek into your world.

Matthew Sullivan

Sorry if this question is too naive, but is the indexing primarily cpu bound, memory bound or i/o bound? Would it be helpful or possible to use a cloud computing to create indexes in a speedy fashion?

I've just heard that slow indexing speed is a major drawback of doc dbs, a prime reason why reporting etc needs to be done on sql... just wondering if you can throw a little cloud money at the problem to get faster turnaround on ad hoc reporting or index fixes.

Ayende Rahien

Matthew, It is using a lot of CPU for full text indexing, it requires a lot of memory and it writes a lot to disk. It wouldn't be workable to do this on the cloud, because the cost of actually sending the data up there and then getting it back would be too high.

Rafal

A nice thing about indexing in Raven is that usually you have all recently modified documents in memory so you can index them without reading from the storage. You will not have such luxury when the lucene index is external to the application

Frank Quednau

Have you implemented your own Task Scheduler for the Task library?

Ayende Rahien

Frank, No, we didn't do that. We handle the control in a much simple concept by partitioning the work before starting the parallel work

Adam Langley

The wonderful thing about that scenario, is if the region of code you are optimising is modular (which I'm sure it is), the problem space is not variable once the software is installed. Hence, you could provide two indexing modules, one designed for single-core, and one for multi-core parallelism.

Of course you would increase your code maintenance, but thats just another decision...

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
  2. Special Offer (2):
    27 May 2015 - 29% discount for all our products
  3. RavenDB Sharding (3):
    22 May 2015 - Adding a new shard to an existing cluster, splitting the shard
  4. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  5. Interview question (2):
    30 Mar 2015 - fix the index
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats