Is select() broken? Memory mapped files with unbufferred writes == race condition?
Let me start this post by stating that I am not even sure if what I am trying to do is legal here. But from reading the docs, it does appear to be a valid use of the API, and it does work, most of the time.
The full code can be found here: https://gist.github.com/ayende/7495987
The gist of it is that I am trying to do two things:
- Write to a file opened with FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING.
- Read from this file using a memory map.
- Occasionally, I get into situations where after I wrote to the file, I am not reading what I wrote.
I have a repro, and we reproduced this on multiple machines. Both Windows 7 and Windows 8.
Here is the relevant code (the full code is in the link), explanation on it below:
This code is doing the following:
- We setup a file handle using NoBuffering | Write_Through, and we also map the file using memory map.
- We write 3 pages (12Kb) at a time to the file.
- After the write, we are using memory map to verify that we actually wrote what we wanted to the file.
- _At the same time_ we are reading from the same memory in another thread.
- Occasionally, we get an error where the data we just wrote to the file cannot be read back.
Now, here is what I think is actually happening:
- When we do an unbuffered write, Windows has to mark the relevant pages as invalid.
- I _think_ that it does so before it actually perform the write.
- If you have another thread that access that particular range of memory at the same time, it can load the _previously_ written data.
- The WriteFile actually perform the write, but the pages that map to that portion of the file have already been marked as loaded.
- At that point, when we use the memory mapped pointer to access the data, we get the data that was there before the write.
As I said, the code above can reproduce this issue (you might have to run it multiple times).
I am not sure if this is something that is valid issue or just me misusing the code. The docs are pretty clear about using regular i/o & memory mapped i/o. The OS is responsible to keeping them coherent with respect to one another. However, that is certainly not the case here.
It might be that I am using a single handle for both, and Windows does less checking when that happens? For what it is worth, I have also tried it using different handles, and I don’t see the problem in the code above, but I have a more complex scenario where I do see the same issue.
Of course, FILE_FLAG_OVERLAPPED is not specified, so what I would actually expect here is serialization of the I/O, according to the docs. But mostly I need a sanity check to tell me if I am crazy.