http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660@Manu Parallel.For is for compute-bound operations. Ayende's example uses IO-bound ones http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment27http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment27Thu, 14 Oct 2010 04:17:55 GMTI am not sure what happens, but why would you not chose to use Paralles.For to achieve the same outcome? http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment26http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment26Wed, 13 Oct 2010 17:17:46 GMTIMO: The correct solution should be based on the separate writing thread which collects data from the other threads and owns the file handler and whole writing and error handling logic. Regards. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment25http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment25Wed, 13 Oct 2010 09:11:14 GMT@Sam - to be honest, I'd have done that to myself! Nick's answer is definitely clearer :-) http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment24http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment24Tue, 12 Oct 2010 22:24:30 GMTOn my computer this gives unpredictable output with every run sometimes missing numbers or entering 0x00 in random places. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment23http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment23Tue, 12 Oct 2010 21:35:23 GMTerr, I mean James & Nick. Sorry James I pulled a TL;DR on your answer :) http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment22http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment22Tue, 12 Oct 2010 21:29:40 GMTMethinks Nick is teh winnar. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment21http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment21Tue, 12 Oct 2010 21:26:39 GMTOn second thought (and rereading the msdn doco on the method), I take that back, FileShare.None only applies to the act of the opening of the stream, not any actual IO done over it. My bad... http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment20http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment20Tue, 12 Oct 2010 19:41:51 GMTActually, I think that the code will work and satisfy the requirements (i.e. noncorrupted, order-insensitive write) due to FileShare.None. All of the tasks will get scheduled on the thread pool, will run in an arbitrary order, but only one at a time will be able to write to the stream, effectively serializing them. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment19http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment19Tue, 12 Oct 2010 19:00:23 GMTSince MSDN help of msdn.microsoft.com/.../system.io.filestream.aspx ">FileStream tells Any public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. it is not safe to use Write() method from different threads, so the above piece of code could actually corrupts the data. Actually with integer number, is less likely you encounter a corruption, but if you write longer string you can easily spot that writes are actually intermixed. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment18http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment18Tue, 12 Oct 2010 17:01:42 GMTYou're going to get some of the numbers being written over, and some will have invalid data (either 3, 4, or 5 bytes of it?) before them. This code inside FileStream is the culprit: Buffer.InternalBlockCopy(array, offset, this._buffer, this._writePos, byteCount); this._writePos += byteCount; If two of the tasks are executing this at almost the same time, one may copy over what the other has just copied to the buffer, but _writePos will still get increased twice (probably, at least). http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment17http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment17Tue, 12 Oct 2010 16:47:53 GMTYou'll get garbled content in the files, but no exceptions. This is not platform/OS dependent because the problem is not with the file writes, but the FileStream internal buffer. You are attempting to write a total of 640 bytes which is a lot less than the default 4K internal buffer for FileStream. So you will not actually write to the file until the Flush(). Thus you will not get any threading issues with accessing the file itself. So the real question is, what happens with multiple threads updating the internal buffer? Reflector shows that both the actual change to the internal buffer, and the change to keep track of the write position are not thread-safe. Both of these could cause corruption of the writes. However they will not cause any exceptions, and so you will have something in the internal buffer to write to the file when Flush() is called, even though it could be corrupted. Caveat: This is for .NET 2.0, not looked into other versions. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment16http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment16Tue, 12 Oct 2010 16:26:09 GMTI just confirmed with reflector that filestream does not synchronize its buffer as I assumed. This means that the program will likely not complete because of a contract exception inside of the filestream. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment15http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment15Tue, 12 Oct 2010 16:08:32 GMT* You could have out of order writes - which you stated isn't a problem. This is pretty likely. * You could have "corrupted" entries - the acess through FileStream.Write isn't syncronous, it uses an internal buffer, and it uses native methods and passes in buffer offsets to do the write. It would be be somewhat easy to have a write "smash" the buffer and end up writing NULL instead of the characters of the correct string, or possibly even duplicate a string. This is fairly likely. * The documentation says IOExceptions could occurr with multiple-thread writing, but it looks like that only happens when the native handle is exposed. I'm not sure this could happen. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment14http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment14Tue, 12 Oct 2010 14:30:01 GMTYes, the big issue here is that file access through a writer is not thread safe at all. Not just the file position pointer, but the actual write buffer as well - fileStream.Write() simply queues up the write and returns. I'm guessing the output of your code is a file with 0-x things written into it, and an IOException. BTW: There is a way to get a synchronized stream for a text writer "TextWriter.Synchronized()", but this requires using a TextWriter instead of a FileStream. It also has the downside of having the FCL be in charge of your performance if this is constantly called code. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment13http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment13Tue, 12 Oct 2010 14:02:11 GMTDoes this blog have a comments feed? http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment12http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment12Tue, 12 Oct 2010 13:44:09 GMT@Lachlan According to [MSDN](http://msdn.microsoft.com/en-us/library/system.threading.tasks.taskscheduler.maximumconcurrencylevel.aspx) MaxConcurrencyLevel is actually Int32.MaxValue. The whole purpose of System.Threading.Tasks is to manage concurrency, not to prevent it. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment11http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment11Tue, 12 Oct 2010 13:07:23 GMTMy guess would be 1 150 149 148 ... 2 http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment10http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment10Tue, 12 Oct 2010 12:30:59 GMTThe default TaskFactory is configured with TaskScheduler,MaximumConcurrencyLevel == 1 http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment9http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment9Tue, 12 Oct 2010 12:24:35 GMTOnly two things can happen: Either whole lines are saved or partial lines can occur. I believe what actually happens depends on the file system implementation. My guess is that whole lines will be preserved although this is not guaranteed. Most speculations in the comments are wrong. No IOException will occur, the file position will not get corrupted and the closure issue has explicitly been avoided. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment8http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment8Tue, 12 Oct 2010 11:52:11 GMT "Since the code block which is executed async is a closure which accesses the outer 'i' variable the code would probably never write the first numbers (see csharpindepth.com/Articles/Chapter5/Closures.aspx)." Hmm.. looks like this local variable prevent such behavior var index = i; http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment7http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment7Tue, 12 Oct 2010 11:37:39 GMTAccording to the MS docs you will likely get an exception because each thread does not have an exclusive lock on its handle: "When a FileStream object does not have an exclusive hold on its handle, another thread could access the file handle concurrently and change the position of the operating system's file pointer that is associated with the file handle... If an unexpected change in the handle position is detected in a call to the Write method, the contents of the buffer are discarded and an IOException is thrown." http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment6http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment6Tue, 12 Oct 2010 11:36:20 GMTSince the code block which is executed async is a closure which accesses the outer 'i' variable the code would probably never write the first numbers (see [csharpindepth.com/Articles/Chapter5/Closures.aspx](http://csharpindepth.com/Articles/Chapter5/Closures.aspx)). IMHO the output would probably be something like 148 148 149 149 150 150 150 ... repeat 150 times http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment5http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment5Tue, 12 Oct 2010 11:23:03 GMTYou will occasionally get the incorrect sequence of binary bits written to the file because multiple threads are writing to the file stream at the same time. To a computer or human interpreting the binary data in the resulting file as ASCII, the symptoms would be missing characters or incorrect characters. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment4http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment4Tue, 12 Oct 2010 11:00:20 GMTI’m pretty sure the file position will get corrupted, meaning that the code will probably overwrite some data (although it may take far more than 150 iterations for the error to manifest), which the CLR may detect and then throw an IO Exception. However, I don’t see the point of potentially launching 150 threads (the thread pool will inject more threads when they block, which is the case with Write since all IO on windows is asynchronous). You should use continuations to serialize the access to the underlying stream and, if you think the cost of asynchronous IO justifies itself (it involves a kernel APC and more context switching), use TaskFactory.FromAsync to build the tasks. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment3http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment3Tue, 12 Oct 2010 10:59:01 GMTIndex variable looks as immutable. Also file flashed at the end of all tasks. So the question is - can something like MemoryStream (some underground stream wich are used for buffering) support simultaneous write access? I don't know exactly but as I remember Console.Output allow writing from different threads. So we probably receive mess of numbers. http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment2http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment2Tue, 12 Oct 2010 10:38:33 GMTfileStream.Write(buffer, 0, buffer.Length) can be interrupted and you end up with corrupted data on disk ? write to disk should be serialized ? http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment1http://ayende.com/4659/riddle-me-this-file-access-multi-threading#comment1Tue, 12 Oct 2010 10:29:19 GMT