http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Sorry, what I meant, is that they aren't short lived, they are put into a processing queue and it may be quite a while until they are processed and some of them discarded http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment14http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment14Mon, 28 Dec 2009 12:07:14 GMTPatrick, They _aren't_ short lived, they go, almost always as-is, to live as part of the in memory model http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment13http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment13Mon, 28 Dec 2009 12:06:00 GMTI am surprised that your program overwhelm the GC by allocating short-living strings. Certainly the GC team would be interested by your experience. I am not sure to understand, what is the content of your strings, code element names? profiler events descripion? http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment12http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment12Mon, 28 Dec 2009 11:46:28 GMTPatrick, I am speaking out of experience here, I just finished doing MAJOR perf session on the profiler. http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment11http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment11Mon, 28 Dec 2009 11:30:40 GMTAre you sure that you can overwhel the GC this way? Internally the GC has its own heap for string and is extremely optimized to allocate/deallocate many strings. When it comes to performance and GC, common sense and suppositions are of poor help. Code it, measure it, and see where is the bottleneck, if any. Many times I had good surprises. And when I had bad surpirses, they never came from where I expected. http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment10http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment10Mon, 28 Dec 2009 07:56:13 GMTPatrick, You are correct, in a sense. The problem is that uncompressing a string will require me to allocate more memory, which leads to the GC having to do more work in the end. My main worry isn't CPU time, it is memory, and having to keep allocating new strings (by the millions, btw) leads to a lod of garbage that the GC have to clean up. http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment9http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment9Mon, 28 Dec 2009 05:00:20 GMTThe problem is to keep many string values alive (i.e not collectable by GC) in memory, so you need a way to compress/uncompress strings values (with a set of words or UTF8 or anything else). String interning is not really compression in the sense that you keep the whole strings values in memory. Strings interning just avoid duplication of same values. Moreover, string interning is not computation free, computing hash code on strings + dico search is not free and even, proposed string hash code impl in .NET Fx are far from being optimal in terms of performance. I think you need something more tricky than string interning to save some memory. >As for UTF8, it doesn't work if you need to translate to standard strings all the time I don't get you, once you need a string value, just uncompress it, use it, and immediately release the reference?! http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment8http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment8Sun, 27 Dec 2009 21:55:07 GMTPatick, That is more or less what I meant by interning strings. As for UTF8, it doesn't work if you need to translate to standard strings all the time, you would allocate a lot of mem just that way http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment7http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment7Sun, 27 Dec 2009 20:15:38 GMTJust an idea that would take a bit of CPU cycles but can drastically reduce memory. In a code base, just a relatively small set of word is used in code elements identifiers. (Do, Go, String, Session, Event, On, Profile, Handler, get_, (, ), User...). You can harness that fact to your advantage by building a set of these words on the fly and a string identifier becomes an array of ids of these words. Another idea: store strings using UTF8, since 99% of char in identifiers are in the first 128 ASCII chars (and take only one byte footprint instead of two). Recently I applied several others tricky-like algo in NDepend with incredible performance/mem consumption improvements. What I discovered in 2009 is that using very tricky ideas, I never heard about in 15 years of programming, pays off. The result is to be released in a few weeks, I cannot wait :o) http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment6http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment6Sun, 27 Dec 2009 19:49:42 GMTNow wondering whether the next series of posts will feature disk based index shards and external sorting... Or perhaps it will be about creating summaries... http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment5http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment5Sun, 27 Dec 2009 06:14:40 GMTWhen OS runs out of memory it swaps. Either you make some sort of clustering (main string + differences) or you swap memory to disk. http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment4http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment4Sat, 26 Dec 2009 19:57:32 GMTYou can provide an option for your tools like "use mmf" and if users encounter with OOME exceptions can try to work in mmf mode. [http://mmf.codeplex.com/](http://mmf.codeplex.com/) you can check this project (memory mapped files) and also .net 4.0 has system.io. memorymappedfiles namespace for this kind of job. btw. luckly, I didn't need to use this projects for now. http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment3http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment3Sat, 26 Dec 2009 12:47:20 GMTMaybe memory-mapped files? http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment2http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment2Sat, 26 Dec 2009 12:18:01 GMTHi Ayende, will you end up with a db for your profiler? http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment1http://ayende.com/4340/the-operation-was-successful-but-the-patient-is-still-dead-deferring-the-obvious-doesn-t-work#comment1Sat, 26 Dec 2009 11:55:34 GMT