Ayende @ Rahien

Refunds available at head office

Machine bias in profiler based optimizations

Take a look at the following profilers results. They both showcase pretty much the same codebase, without any major changes in between. However, they have been both run on different machines, and they result in very different performance optimization paths.

Main machine:

Laptop:

image

As you can see, when running the code on my main machine, the most expensive thing is actually writing to disk (WriteToJournal). However, on my laptop, writing to disk is actually very cheap.  The likely reason is, again, my laptop is faking writing to disk in favor of buffering writes, even though we explicitly tell it not to. That means that it behaves as if it is we already optimized writing to disk. My main machine behaves more like a server, and we see a lot more I/O costs, which pretty much mask any other costs that we might have.

Using the profiler to guide you with regards to how to optimize the system on any one of those machines would lead you in very different paths. With Voron, we are testing it in a variety of scenarios, to make sure that we aren’t optimizing for one particular machine and hurting another.

Comments

Vadi
12/30/2013 12:06 PM by
Vadi

Couldn't quite follow how your laptop is "faking" as if it is buffering writes. Are you running different configurations ? Does it right to assume the server is probably talking to NAS.

Greg young
12/30/2013 12:16 PM by
Greg young

"Even though we explicitly tell it not to"...

I understand from your posts you are using o-direct. Using direct io (unbuffered io) does not explicitly say to the disk not to cache. It tells the file system not to cache. I have mentioned this before but you have to turn off caching in windows. That little check box does not just control windows caching, if enabled it also turns on caching on your controller. You can see this easily by grabbing a USB stick with linux on it and using hdparm. You can enable it if using disk specific tools to ensure that the controller caching is disabled.

Further it gets more complicated than this. Many commodity grade disks will still cache (and reorder writes) even if you explicitly turn off their caching! They just ignore the fact that you told them. Benchmarking disk io is hard :)

Ayende Rahien
12/30/2013 06:16 PM by
Ayende Rahien

Vadi, I am not sure how to characterize it better. It is drastically faster than it should, even when we tell it not to.

Ayende Rahien
12/30/2013 06:17 PM by
Ayende Rahien

Greg, Yes, I ,know. I have done that, and I still see perf that is way above what the disk is supposed to be able to give me. That is why I said that I believe it is faking it.

Ayende Rahien
12/30/2013 06:18 PM by
Ayende Rahien

Greg, This is also why we make certain assumptions about our disks, and document them. We can't work around what they will do if they explicitly violate their own docs.

Greg Young
12/30/2013 06:19 PM by
Greg Young

The real question you need to ask is if it has a super cap on it. If it does what it's doing is totally valid!

Daniel Marbach
12/30/2013 10:18 PM by
Daniel Marbach

He means super capacitors which support write coalescing

http://thessdguy.com/how-controllers-maximize-ssd-life-external-data-buffering/#more-628

Comments have been closed on this topic.