Ayende @ Rahien

Ayende Rahien commented on String Interning: The Garbage Collectible way

Wed, 06 Jan 2010 22:03:27 GMT

Hernique, I would NEVER do something like that

Henrique commented on String Interning: The Garbage Collectible way

Wed, 06 Jan 2010 22:00:20 GMT

There is a problem in this case, if you intend to have only one string reference for a given string, you will fail. if you do something like this, for example: lock(String.Intern(product.Id)) { } The "Clear" method will break this kind of trick. Perhaps, WeakReference would be useful here

Rafal commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 22:26:36 GMT

@Leon & others It doesn't make sense to 'scroll' through the buffer of text messages if the buffer contains a gigabyte of data or more. You won't find anything by just scrolling, you need a search engine... So, it means keeping the raw text in memory doesn't make sense, much better to keep just the index in memory and retrieve raw text only when drilling down into details. @Ayende, My colleague has recently built an OLAP cube on IIS log files and uses that cube to analyze the application behavior. It was astonishing how much information he retrieved - he was able to create application performance graphs, analyze activity of users, identify application URLS that have too long execution time or transfer too much data, find errors in integration interfaces, identify several bottlenecks, everything by just slicing & dicing the cube.. Maybe something similar would be useful in your profiler? Statistical analysis is a very effective profiling method especially when you have to analyze large sets of data and today we are talking about amounts like millions of rows of data per day. We are using the OLAP as a capacity management tool - we process large amounts of information (think months of iis logs) and analyze the patterns and trends that emerge over time so we have a chance to identify performance problems before they are noticed by customers.

Leon Breedt commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 21:43:58 GMT

Have you considered a fixed-size buffer of strings, backed by disk storage? e.g. circular buffer of some sort. Since I'm assuming your massive collection is append only (e.g. no-one comes and removes a string from the middle). Your disk storage would then be indexed by line number, as would entries in your buffer....Someone tries to scroll outside of the starting and finishing bounds of your buffer, and you load in as many strings as appropriate from storage. You'll want to optimize the disk storage for reading obviously, if you're using some form of list virtualization, you would probably want to have a mapping between line number and offset in disk file so you can do a Seek() to starting line and start reading. Constrained memory usage, at the expense of a bit of a lag when scrolling far outside the current bounds.

Mike Chaliy commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 21:42:48 GMT

+1 to Dmitry, for example Sqlite

Dmitry commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 18:59:25 GMT

Have you decided to persist strings to a temp storage instead of keeping them in memory?

Ayende Rahien commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 17:10:50 GMT

Jon, Because when you do perf testing, you want to make as few changes as you possibly could, fixing one problem at a time. The worst thing you can do is to try to patch over the root problem

Jon commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 16:46:41 GMT

Why not leave in the string interning code? Even if you do only delay the eventual a 50% decrease in memory sounds like a great thing.

Ayende Rahien commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 14:43:40 GMT

Ken, Because I there is no way with a HashSet to get the value matching, and I care about the particular reference that I use.

Ayende Rahien commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 14:42:59 GMT

Imran, That would keep allocating new strings every time I do the conversion back. I might use less memory in total, but I would allocate a lot more (more GC work)

Ken Egozi commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 13:50:24 GMT

why Dict and not HashSet

Imran commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 11:25:01 GMT

Another thing you can add to reduce the memory footprint further is utf8 encoding. Store the strings as a byte[] instead by calling Encoding.UTF8.GetBytes(string) to circumvent the default .net utf16 encoding. You can use Encoding.UTF8.GetString to convert back to string as needed. Some characters will still be encoded with 2 bytes but for most ascii strings this will give a memory benefit. As you said before though, it doesn't solve your problem it just delays it.

Ayende Rahien commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 11:16:25 GMT

Robert, The short answer is that EnterUpgradeableReadLock would produce a far higher locking rate than read/write.

Ayende Rahien commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 11:05:18 GMT

Robert, I'll answer that in a separate post.

Robert commented on String Interning: The Garbage Collectible way

Sun, 27 Dec 2009 10:38:17 GMT

Any reason for which you don't use EnterUpgradeableReadLock in Intern method?