http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Tapio, Yes, of course. Please see the previous post about this topic http://ayende.com/4829/ravenfs-rsync#comment13http://ayende.com/4829/ravenfs-rsync#comment13Sat, 07 May 2011 08:50:48 GMTHave you considered "deduplication". It's an interesting idea and It could speed up synchronization of some files. It won't help with video files unless the FS or the receiving system already has a copy of the file somewhere else. [highscalability.com/.../...ical-deduplication.html](http://highscalability.com/blog/2011/5/5/paper-a-study-of-practical-deduplication.html)http://ayende.com/4829/ravenfs-rsync#comment12http://ayende.com/4829/ravenfs-rsync#comment12Fri, 06 May 2011 18:45:35 GMTI guess it would depend on closely analyzing the pattern of modification of typical files the application would be serving. So long as files that use header information have a relatively static sized header (so that changes to the file size don't mean"early" page sizes are invalidated.) The technology also lends itself to peer-to-peer updates where clients can query not only the master server for the most current pages, but each other as well to take advantage of available bandwidth. http://ayende.com/4829/ravenfs-rsync#comment11http://ayende.com/4829/ravenfs-rsync#comment11Tue, 03 May 2011 08:58:32 GMTAre you considering using it to create, sorry for the crappy term but a dropbox-killer? http://ayende.com/4829/ravenfs-rsync#comment10http://ayende.com/4829/ravenfs-rsync#comment10Tue, 03 May 2011 01:02:04 GMTis this more similar to zsync then : [http://zsync.moria.org.uk/](http://zsync.moria.org.uk/)http://ayende.com/4829/ravenfs-rsync#comment9http://ayende.com/4829/ravenfs-rsync#comment9Mon, 02 May 2011 23:03:43 GMTI think that with all the press that scalability is getting with all the various DBs and file systems from Amazon, Facebook, Google, MongoDB, etc. that this issue is not a solved one. So either they do have a solution and aren't sharing, or it's so hard that they can't solve it well enough for general case. Watching a one man team beat them would be fun, if nothing else :) http://ayende.com/4829/ravenfs-rsync#comment8http://ayende.com/4829/ravenfs-rsync#comment8Mon, 02 May 2011 18:49:19 GMTSchooletz, I haven't considered this, but it certainly can, in a one way fashion. http://ayende.com/4829/ravenfs-rsync#comment7http://ayende.com/4829/ravenfs-rsync#comment7Mon, 02 May 2011 11:26:03 GMTWill the RavenFS be used as a synchronization tool for some db with an append only model? It looks like a perfect match. http://ayende.com/4829/ravenfs-rsync#comment6http://ayende.com/4829/ravenfs-rsync#comment6Mon, 02 May 2011 10:36:44 GMTI would pay for a tool like this...but has to be simple to use and efficient! http://ayende.com/4829/ravenfs-rsync#comment5http://ayende.com/4829/ravenfs-rsync#comment5Mon, 02 May 2011 07:05:50 GMTJohn, Yes, I have. We had customers complain about issues with those. I suspect that it had to do with the complexity involving security, domains, configuration, etc. http://ayende.com/4829/ravenfs-rsync#comment4http://ayende.com/4829/ravenfs-rsync#comment4Mon, 02 May 2011 06:26:06 GMTAyende, Have u consider using DFSR? It is part of Windows 2003 R2 and above OS. It works quite well on crappy WAN links and it does diffs + compression. See, [msdn.microsoft.com/.../bb540025%28v=vs.85%29.aspx](http://msdn.microsoft.com/en-us/library/bb540025%28v=vs.85%29.aspx) and [en.wikipedia.org/.../Distributed_File_System_%2...](http://en.wikipedia.org/wiki/Distributed_File_System_%28Microsoft%29) for more details http://ayende.com/4829/ravenfs-rsync#comment3http://ayende.com/4829/ravenfs-rsync#comment3Sun, 01 May 2011 23:43:03 GMTI belive you considered using torrent server. Why aren't torents an option? http://ayende.com/4829/ravenfs-rsync#comment2http://ayende.com/4829/ravenfs-rsync#comment2Sun, 01 May 2011 17:43:03 GMTI implemented a backup tool that works like rsync for fun myself. I noticed only in hindsight that either the whole file changes entirely (docx, zip, avi, ... no partial changes) or it changes in blocks (database, truecrypt). Thats for exactly the reason you gave - it is to expensive for the application itself to do it otherwise. You pragmatism is what makes this blog interesting. You just jettisoned a part of the problem that is of high cost and of low value by not allowing insertions. Its good enough. http://ayende.com/4829/ravenfs-rsync#comment1http://ayende.com/4829/ravenfs-rsync#comment1Sun, 01 May 2011 13:34:41 GMT