http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Enjoyed a cup of coffeehttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment24http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment24Wed, 28 Mar 2012 08:42:03 GMTMemory-map the files?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment23http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment23Tue, 27 Mar 2012 21:21:22 GMTDisabled your anti-virus?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment22http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment22Tue, 27 Mar 2012 20:37:50 GMTDefraghttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment21http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment21Tue, 27 Mar 2012 18:55:32 GMTYou told the freedb team to get their shit together and clean their mess up?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment20http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment20Tue, 27 Mar 2012 18:50:51 GMTLoaded them into RavenFS ;-)http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment19http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment19Tue, 27 Mar 2012 17:57:41 GMTAs you probably already had SSD, you did nothinghttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment18http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment18Tue, 27 Mar 2012 16:33:58 GMTPut the kettle on? Every problem is a magnitude simpler when tackled with a hot cup of coffee.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment17http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment17Tue, 27 Mar 2012 14:42:48 GMTIncreased the file chunk size to match the NTFS storage format chunk sizehttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment16http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment16Tue, 27 Mar 2012 14:11:51 GMTYou had done a regular data import and RavenDB did its magic.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment15http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment15Tue, 27 Mar 2012 14:02:10 GMTSSDhttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment14http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment14Tue, 27 Mar 2012 13:40:42 GMTYeah, he tried with no luck. I agreed to providing mental support at 1AM instead, though. :-)http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment13http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment13Tue, 27 Mar 2012 13:18:40 GMTLike flukus said... or even better, mounted a ram disk and filled it with your 3.1 million files.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment12http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment12Tue, 27 Mar 2012 13:16:04 GMT+1 for Itamar. Handballing is an extremely efficient operation.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment11http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment11Tue, 27 Mar 2012 13:07:11 GMTHandball the problem over to Itamar?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment10http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment10Tue, 27 Mar 2012 11:42:06 GMTMove the files to SSD?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment9http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment9Tue, 27 Mar 2012 11:38:41 GMTMy two cents: Use IO completion ports ( async ) for reading the files, but mantain the reading strictly sequential. In the "done" function publish an job and consume it by another ( more than one? thread(s) ) pushing data into the target, this will at least compensate the seek time by doing something useful, and possibly if somethime the push opertaion is delayed ( I guess it can be, but maybe I'm wrong ) you can even keep scanning the hard drive. http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment8http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment8Tue, 27 Mar 2012 11:00:47 GMTI found some earlier talk about this in google groups 8 feb. Looking a github, you added Self optimizing on batch-sizes: https://github.com/ravendb/ravendb/commit/294c2134c5fa7b0b95d0297dfac38cb9ab9acd38http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment7http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment7Tue, 27 Mar 2012 10:39:21 GMTBatching Disable indexing (POST /admin/stopindexing )http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment6http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment6Tue, 27 Mar 2012 10:25:18 GMTI would say to merge the files. Makes it easier for subsequent work. But this merge itself would take some time. But this is really good place for asynchronous IO.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment5http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment5Tue, 27 Mar 2012 10:13:42 GMTCompensate for the seek time somehow? Assuming you were on spinning metal driveshttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment4http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment4Tue, 27 Mar 2012 10:09:53 GMTmerge filehttp://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment3http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment3Tue, 27 Mar 2012 10:09:24 GMTRun it in the profiler.http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment2http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment2Tue, 27 Mar 2012 10:07:36 GMTgzip all files and then read the compressed stream at once?http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment1http://ayende.com/154433/watch-your-6-or-is-it-your-i-o#comment1Tue, 27 Mar 2012 10:07:03 GMT