Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 12 | Comments: 44

filter by tags archive

Machine bias in profiler based optimizations

time to read 2 min | 228 words

Take a look at the following profilers results. They both showcase pretty much the same codebase, without any major changes in between. However, they have been both run on different machines, and they result in very different performance optimization paths.

Main machine:



As you can see, when running the code on my main machine, the most expensive thing is actually writing to disk (WriteToJournal). However, on my laptop, writing to disk is actually very cheap.  The likely reason is, again, my laptop is faking writing to disk in favor of buffering writes, even though we explicitly tell it not to. That means that it behaves as if it is we already optimized writing to disk. My main machine behaves more like a server, and we see a lot more I/O costs, which pretty much mask any other costs that we might have.

Using the profiler to guide you with regards to how to optimize the system on any one of those machines would lead you in very different paths. With Voron, we are testing it in a variety of scenarios, to make sure that we aren’t optimizing for one particular machine and hurting another.



Couldn't quite follow how your laptop is "faking" as if it is buffering writes. Are you running different configurations ? Does it right to assume the server is probably talking to NAS.

Greg young

"Even though we explicitly tell it not to"...

I understand from your posts you are using o-direct. Using direct io (unbuffered io) does not explicitly say to the disk not to cache. It tells the file system not to cache. I have mentioned this before but you have to turn off caching in windows. That little check box does not just control windows caching, if enabled it also turns on caching on your controller. You can see this easily by grabbing a USB stick with linux on it and using hdparm. You can enable it if using disk specific tools to ensure that the controller caching is disabled.

Further it gets more complicated than this. Many commodity grade disks will still cache (and reorder writes) even if you explicitly turn off their caching! They just ignore the fact that you told them. Benchmarking disk io is hard :)

Ayende Rahien

Vadi, I am not sure how to characterize it better. It is drastically faster than it should, even when we tell it not to.

Ayende Rahien

Greg, Yes, I ,know. I have done that, and I still see perf that is way above what the disk is supposed to be able to give me. That is why I said that I believe it is faking it.

Ayende Rahien

Greg, This is also why we make certain assumptions about our disks, and document them. We can't work around what they will do if they explicitly violate their own docs.

Greg Young

The real question you need to ask is if it has a super cap on it. If it does what it's doing is totally valid!

Daniel Marbach

He means super capacitors which support write coalescing


Comment preview

Comments have been closed on this topic.


  1. Production postmortem: The case of the lying configuration file - one day from now
  2. Production postmortem: The industry at large - about one day from now
  3. The insidious cost of allocations - 3 days from now
  4. Find the bug: The concurrent memory buster - 4 days from now
  5. Buffer allocation strategies: A possible solution - 7 days from now

And 3 more posts are pending...

There are posts all the way to Sep 10, 2015


  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    31 Aug 2015 - The case of the memory eater and high load
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats