http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Sorry, I found the answer in your previous post :). Blogs shows latest first, so I was reading this one before the persistence one :) http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment18http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment18Wed, 30 Dec 2009 05:06:00 GMTJust a question : Since you are serializing huge amount of data and then filtering, couldn't you use some embedded Database like SQLite ? I have used it in some situations like this and found it to be useful. But to be honest, no idea of the challenges you are dealing here, so i may be wrong. http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment17http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment17Wed, 30 Dec 2009 04:56:32 GMTI remember something about a fast serialization approach developed by Greg Young. Would this be helpful to you? [codebetter.com/.../fast-serialization.aspx](http://codebetter.com/blogs/gregyoung/archive/2008/08/24/fast-serialization.aspx)http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment16http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment16Tue, 29 Dec 2009 23:26:43 GMTHi What about T4 to generate your serialization code, may be easier to maintain? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment15http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment15Tue, 29 Dec 2009 20:54:45 GMTAnon, That isn't an issue. I am only loading chunks (a single session) from the file, so unless I have a single _session_ that consume more than the available memory, it is a non issue. And if I do, well, there are other problems to be solved there first. http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment14http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment14Tue, 29 Dec 2009 17:56:27 GMTAre you planning to handle the scenario of a initial file that is larger than total available memory for these changes? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment13http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment13Tue, 29 Dec 2009 16:31:54 GMTAh! Great! Of coarse. I was feeling rather lowly being the bearer of bad news after such great progress... http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment12http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment12Tue, 29 Dec 2009 16:05:12 GMTRob, There is always pre-build step :-) http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment11http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment11Tue, 29 Dec 2009 16:03:25 GMTFrans, I did. It did a lot of things manually, and that wasn't what I was after. http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment10http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment10Tue, 29 Dec 2009 16:03:06 GMTOk. This is really going to frustrate you. But, there is not CodeDom in Silverlight ;( There is Reflection.Emit though... http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment9http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment9Tue, 29 Dec 2009 16:01:48 GMTYou didn't look at the fast serialization articles I mentioned to you ? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment8http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment8Tue, 29 Dec 2009 14:36:38 GMTInteresting stuff, would be interested in seeing some of the code you have used to do this. Any chance of posting part of it? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment7http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment7Tue, 29 Dec 2009 12:24:08 GMTJulien, I _am_ making use of Google Buffer already. The problem is that it requires me to adhere to a very rigid structure, which would limit my ability to work with it http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment6http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment6Tue, 29 Dec 2009 12:22:02 GMTHave you look into google's protocols buffers? it's a lot faster and space efficient than BinarryFormatter : [code.google.com/p/protobuf-net/wiki/Performance](http://code.google.com/p/protobuf-net/wiki/Performance)http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment5http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment5Tue, 29 Dec 2009 12:19:16 GMTZamboch, When deserializing, the data is pretty small,so I never actually had to check that. http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment4http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment4Tue, 29 Dec 2009 12:15:56 GMTWhen deserializing, how much of the time take memory allocation itself ? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment3http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment3Tue, 29 Dec 2009 12:13:37 GMTSome of the data is thrown away, or kept in a much more compact form. http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment2http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment2Tue, 29 Dec 2009 12:05:59 GMTHow comes that serializing 900 Mb of messages gives a 454Mb file? http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment1http://ayende.com/4343/fighting-the-profiler-memory-obesity#comment1Tue, 29 Dec 2009 12:04:43 GMT