http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Having such huge amount of file is a pain for every filesystem, because you have files that are smaller than the size of a cluster in the disk, and this mean wasted space and wasted time in reading and writing on disk. The picture of ayende shows how the real file size on disk is four time bigger than file size. Having them stored as a single compressed file is a good strategy, because you are trading disk activity with CPU activity to decompress the file and this is a gain because disk is much more slower. The good part is that all compression libraries have the option to create an archive at 0 compression level, this permits to avoid spending CPU cycle for decompression, but with the advantage of handling everything as a single file. http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment9http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment9Fri, 06 Apr 2012 07:47:23 GMT@Dan, Patrick, Darius: Are you suggesting that Ayende should tell all his customers to install SSDs on their DB servers instead of enabling his software to read tarballs?http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment8http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment8Fri, 30 Mar 2012 08:03:12 GMT>>How about just getting a good SSD? :D How about getting a decent file system?http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment7http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment7Thu, 29 Mar 2012 15:23:43 GMT@Dan: +1 Because if Ayende had gotton a good SSD, the Size on disk of all his files would've been less than 743 MB.http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment6http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment6Thu, 29 Mar 2012 07:24:04 GMTYes, it's a shame modern 'high performance' filesystems (NTFS, i'm talking to you) can't handle 50 thousand small files in a single directory. Some time ago a bug in my company's application that resulted in creation of thousands of temp files brought down the whole server - IIS, the application and even file explorer. And it was hard to find too because the filesystem behaved as if the hardware was failing or the disk logical structure was corrupt.http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment5http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment5Wed, 28 Mar 2012 18:00:38 GMTHow about just getting a good SSD? :Dhttp://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment4http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment4Wed, 28 Mar 2012 17:53:27 GMT@Steve: ext* would still be a lot slower to read millions of files in sequence than using an archive, assuming you don't care about the order in which you read files. It's because a file's size is measured in pages, and you get lots of free space between the files, so even if the files are all sequential on disk there's a lot of seek time. I think even an uncompressed tar file would be faster than pretty much any OS. Not to mention that most compression schemes (e.g. zip, gzip, but not bz2) can be decompressed faster than they are read from disk on modern machines, meaning I/O is still the bottleneck even on compressed files.http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment3http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment3Wed, 28 Mar 2012 15:53:32 GMTMy thought is that NTFS just plain sucks with that much files even if they are large. UNIX filesystem dose not have that much problems with it.http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment2http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment2Wed, 28 Mar 2012 15:31:42 GMTDamn, I thought about that too. Now I'm sad I didn't mention it in previous thread.http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment1http://ayende.com/154465/watch-your-6-or-is-it-your-i-o-it-is-the-i-o-yes#comment1Wed, 28 Mar 2012 10:17:37 GMT