Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,130 | Comments: 45,556

filter by tags archive

Windows Overlapped I/O and TPL style programming

time to read 23 min | 4490 words

I really like the manner in which C# async tasks work. And while building Voron, I run into a scenario in which I could really make use of Windows async API. This is exposed via the Overlapped I/O. The problem is that those are pretty different models, and they don’t appear to want to play together very nicely.

Since I don’t feel like having those two cohabitate in my codebase, I decided to see if I could write a TPL wrapper that would provide nice API on top of the underlying Overlapped I/O implementation.

Here is what I ended up with:

   1: public unsafe class Win32DirectFile : IDisposable
   2: {
   3:     private readonly SafeFileHandle _handle;
   5:     public Win32DirectFile(string filename)
   6:     {
   7:         _handle = NativeFileMethods.CreateFile(filename,
   8:             NativeFileAccess.GenericWrite | NativeFileAccess.GenericWrite, NativeFileShare.None, IntPtr.Zero,
   9:             NativeFileCreationDisposition.CreateAlways,
  10:             NativeFileAttributes.Write_Through | NativeFileAttributes.NoBuffering | NativeFileAttributes.Overlapped, IntPtr.Zero);
  12:         if (_handle.IsInvalid)
  13:             throw new Win32Exception();
  15:         if(ThreadPool.BindHandle(_handle) == false)
  16:             throw new InvalidOperationException("Could not bind the handle to the thread pool");
  17:     }

Note that I create the file with overlapped enabled, as well as write_through & no buffering (I need them for something else, not relevant for now).

It it important to note that I bind the handle (which effectively issue a BindIoCompletionCallback under the cover, I think), so we won’t have to use events, but can use callbacks. This is much more natural manner to work when using the TPL.

Then, we can just issue the actual work:

   1: public Task WriteAsync(long position, byte* ptr, uint length)
   2: {
   3:     var tcs = new TaskCompletionSource<object>();
   5:     var nativeOverlapped = CreateNativeOverlapped(position, tcs);
   7:     uint written;
   8:     var result = NativeFileMethods.WriteFile(_handle, ptr, length, out written, nativeOverlapped);
  10:     return HandleResponse(result, nativeOverlapped, tcs);
  11: }

As you can see, all the actual details are handled in the helper functions, we can just run the code we need, passing it the overlapped structure it requires. Now, let us look at those functions:

   1: private static NativeOverlapped* CreateNativeOverlapped(long position, TaskCompletionSource<object> tcs)
   2: {
   3:     var o = new Overlapped((int) (position & 0xffffffff), (int) (position >> 32), IntPtr.Zero, null);
   4:     var nativeOverlapped = o.Pack((code, bytes, overlap) =>
   5:     {
   6:         try
   7:         {
   8:             switch (code)
   9:             {
  10:                 case ERROR_SUCCESS:
  11:                     tcs.TrySetResult(null);
  12:                     break;
  13:                 case ERROR_OPERATION_ABORTED:
  14:                     tcs.TrySetCanceled();
  15:                     break;
  16:                 default:
  17:                     tcs.TrySetException(new Win32Exception((int) code));
  18:                     break;
  19:             }
  20:         }
  21:         finally
  22:         {
  23:             Overlapped.Unpack(overlap);
  24:             Overlapped.Free(overlap);
  25:         }
  26:     }, null);
  27:     return nativeOverlapped;
  28: }
  30: private static Task HandleResponse(bool completedSyncronously, NativeOverlapped* nativeOverlapped, TaskCompletionSource<object> tcs)
  31: {
  32:     if (completedSyncronously)
  33:     {
  34:         Overlapped.Unpack(nativeOverlapped);
  35:         Overlapped.Free(nativeOverlapped);
  36:         tcs.SetResult(null);
  37:         return tcs.Task;
  38:     }
  40:     var lastWin32Error = Marshal.GetLastWin32Error();
  41:     if (lastWin32Error == ERROR_IO_PENDING)
  42:         return tcs.Task;
  44:     Overlapped.Unpack(nativeOverlapped);
  45:     Overlapped.Free(nativeOverlapped);
  46:     throw new Win32Exception(lastWin32Error);
  47: }

The complexity here is that we need to handle 3 cases:

  • Successful completion
  • Error (no pending work)
  • Error (actually success, work is done in an async manner).

But that seems to be working quite nicely for me so far.


Justin Van Patten

Very nice. Presumably you'll be calling WriteAsync quite often, creating lots of Task objects. All those allocations will add pressure to the GC. .NET's Socket APIs added a specialized async pattern specifically to avoid the APM's IAsyncResult allocations.

Stephen Toub wrote about how to wrap these Socket APIs in a way that allows them to be awaitable, without the Task allocations. http://blogs.msdn.com/b/pfxteam/archive/2011/12/15/10248293.aspx

I wonder if an approach along these lines would be useful for Voron.

Ayende Rahien

Justin, That is something we'll be looking at, yes. We need to do a lot more stuff first.

Comment preview

Comments have been closed on this topic.


  1. How to waste CPU and kill your disk by scaling 100 million inefficiently - one day from now
  2. RavenDB Conference 2016–Slides - about one day from now

There are posts all the way to Jun 01, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats