I build databases for a living, and as such, I spend a great deal of time working with file I/O. Since the database I build is cross-platform, I run into different I/O behavior on different operating systems all the time.

One of the more annoying aspects for a database developer is handling file metadata changes between Windows and Linux (and POSIX in general). You can read more about the details in this excellent post by Dan Luu.

On Windows, the creation of a new file is a reliable operation.If the operation succeeds, the file exists. Note that this is distinct from when you write data to it, which is a whole different topic. The key here is that file creation, size changes, and renames are things that you can rely on.

On Linux, on the other hand, you also need to sync the parent directory (potentially all the way up the tree, by the way). The details depend on what exact file system you have mounted and exactly which flags you are using, etc.

This difference in behavior between Windows and Linux is probably driven by the expected usage, or maybe the expected usage drove the behavior. I guess it is a bit of a chicken-and-egg problem.

It’s really common in Linux to deal with a lot of small files that are held open for a very short time, while on Windows, the recommended approach is to create file handles on an as-needed basis and hold them.

The cost of CreateFile() on Windows is significantly higher than open() on Linux. On Windows, each file open will typically run through a bunch of filters (antivirus, for example), which adds significant costs.

Usually, when this topic is raised, the main drive is that Linux is faster than Windows. From my perspective, the actual issue is more complex. When using Windows, your file I/O operations are much easier to reason about than when using Linux. The reason behind that, mind you, is probably directly related to the performance differences between the operating systems.

In both cases, by the way, the weight of legacy usage and inertia means that we cannot get anything better these days and will likely be stuck with the same underlying issues forever.

Can you imagine what kind of API we would have if we had a new design as a clean slate on today’s hardware?