Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,026 | Comments: 44,842

filter by tags archive

Machine bias in profiler based optimizations

time to read 2 min | 228 words

Take a look at the following profilers results. They both showcase pretty much the same codebase, without any major changes in between. However, they have been both run on different machines, and they result in very different performance optimization paths.

Main machine:



As you can see, when running the code on my main machine, the most expensive thing is actually writing to disk (WriteToJournal). However, on my laptop, writing to disk is actually very cheap.  The likely reason is, again, my laptop is faking writing to disk in favor of buffering writes, even though we explicitly tell it not to. That means that it behaves as if it is we already optimized writing to disk. My main machine behaves more like a server, and we see a lot more I/O costs, which pretty much mask any other costs that we might have.

Using the profiler to guide you with regards to how to optimize the system on any one of those machines would lead you in very different paths. With Voron, we are testing it in a variety of scenarios, to make sure that we aren’t optimizing for one particular machine and hurting another.



Couldn't quite follow how your laptop is "faking" as if it is buffering writes. Are you running different configurations ? Does it right to assume the server is probably talking to NAS.

Greg young

"Even though we explicitly tell it not to"...

I understand from your posts you are using o-direct. Using direct io (unbuffered io) does not explicitly say to the disk not to cache. It tells the file system not to cache. I have mentioned this before but you have to turn off caching in windows. That little check box does not just control windows caching, if enabled it also turns on caching on your controller. You can see this easily by grabbing a USB stick with linux on it and using hdparm. You can enable it if using disk specific tools to ensure that the controller caching is disabled.

Further it gets more complicated than this. Many commodity grade disks will still cache (and reorder writes) even if you explicitly turn off their caching! They just ignore the fact that you told them. Benchmarking disk io is hard :)

Ayende Rahien

Vadi, I am not sure how to characterize it better. It is drastically faster than it should, even when we tell it not to.

Ayende Rahien

Greg, Yes, I ,know. I have done that, and I still see perf that is way above what the disk is supposed to be able to give me. That is why I said that I believe it is faking it.

Ayende Rahien

Greg, This is also why we make certain assumptions about our disks, and document them. We can't work around what they will do if they explicitly violate their own docs.

Greg Young

The real question you need to ask is if it has a super cap on it. If it does what it's doing is totally valid!

Daniel Marbach

He means super capacitors which support write coalescing


Comment preview

Comments have been closed on this topic.


No future posts left, oh my!


  1. Technical observations from my wife (3):
    13 Nov 2015 - Production issues
  2. Production postmortem (13):
    13 Nov 2015 - The case of the “it is slow on that machine (only)”
  3. Speaking (5):
    09 Nov 2015 - Community talk in Kiev, Ukraine–What does it take to be a good developer
  4. Find the bug (5):
    11 Sep 2015 - The concurrent memory buster
  5. Buffer allocation strategies (3):
    09 Sep 2015 - Bad usage patterns
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats