If you throttle me any me I am going to throttle you back!
It is interesting to note that for a long while, what we were trying to do with RavenDB was make it use less and less resources. One of the reasons for that is that less resources is obviously better, because we aren’t wasting anything.
The other reason is that we have users running us on a 512MB/650 MHz Celeron 32 bit machines. So we really need to be able to fit into a small box (and also allow enough processing power for the user to actually do something with the machine).
We have gotten really good in doing that, actually.
The problem is that we also have users running RavenDB on standard server hardware (32 GB / 16 cores, RAID and what not) in which case they (rightly) complain that RavenDB isn’t actually using all of their hardware.
Now, being conservative about resource usage is generally good, and we do have the configuration in place which can tell RavenDB to use more memory. It is just that this isn’t polite behavior.
RavenDB in most cases shouldn’t require anything special for you to run, we want it to be truly a zero admin database. The solution? Take into account the system state and increase the amount of work that we do to get things done. And yes, I am aware of the pitfalls.
As long as there is enough free RAM available, we will increase the amount of documents that we are going to index in a single batch. That is subject to some limits (for example, if we just created a new index on a big database, we need to make sure we aren’t trying to load it entirely to memory), and it knows how to reserve some room for other things, and how to throttle down and as well as up.
This post is written before I had the chance to actually test this on production level size dataset, but I am looking forward to seeing how it works.
Update: Okay, that is encouraging, it looks like what we did just made things over 7 times faster. And this isn’t a micro benchmark, this is when you throw this on a multi GB database with full text search indexing.
Next, we need to investigate what we are going to do about multiple running indexes and how this optimization affects them. Fun .