Ayende @ Rahien

It's a girl

Investigating an OutOfMemoryException

Originally posted at 1/5/2011

I finally got a reliable reproduction for a repeated error, but I have to say, just based on the initial impression, something very strange is going on.

image

The stack trace was a bit more interesting:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
  at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
  at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)
  at System.Text.StringBuilder.Append(String value)
  at System.Text.RegularExpressions.RegexReplacement.ReplacementImpl(StringBuilder sb, Match match)
  at System.Text.RegularExpressions.RegexReplacement.Replace(Regex regex, String input, Int32 count, Int32 startat)
  at System.Text.RegularExpressions.Regex.Replace(String input, String replacement, Int32 count, Int32 startat)
  at System.Text.RegularExpressions.Regex.Replace(String input, String replacement)
  at System.Text.RegularExpressions.Regex.Replace(String input, String pattern, String replacement)
  at HibernatingRhinos.Profiler.BackEnd.SqlStatementProcessor.ReplaceParametersWithValues(String statement, Boolean useComment) 

I put a breakpoint in the appropriate place, and discovered that the error occurred in:

  • A SQL Statement that was 190 kilobytes in size
  • It had 4,060 parameters

Now, let us look at the actual code:

private string ReplaceParametersWithValues(string statement, bool useComment)
{
  if (sqlStatement.Parameters == null)
    return statement;

  foreach (var parameter in sqlStatement.Parameters
            .Where(x => x != null && x.Name.Empty() == false)
            .OrderByDescending(x => x.Name.Length))
  {
    var patternNameSafeForRegex = Regex.Escape(parameter.Name);
    var pattern = patternNameSafeForRegex + @"(?![\d|_])";
    //static Regex methods will cache the regex, so we don't need to worry about that
      var replacement = useComment ? 
                parameter.Value + " /* " + parameter.Name + " */" : 
                parameter.Value;
      statement = Regex.Replace(statement,
                  pattern,
                  replacement);
  }
  return statement;
}

The problem is that:

  • There is no heavy memory pressure.
  • While the string is big, it is not that big.
  • In practice, there is a single replacement for each parameter.

Just for fun, I wasn’t able to reproduce the issue without running the full NH Prof application.

I solved the issue, but I am not entirely pleased with the way I solved it. (That is tomorrow’s post)

Any ideas how to reproduce this?

Any elegant ideas on how to solve this?

Comments

tobi
01/05/2011 03:08 PM by
tobi

My guess is heap fragmentation due to forced usage of the LOH (big strings and maybe some internal regex objects). But that should still not be a problem (maybe the heap is already very fragmented because the NHProf app is running).

Tim
01/05/2011 03:23 PM by
Tim

The StringBuilder needs a contiguous block of memory for each Append call it makes (if it needs to create a new block of memory - in other worse, when its buffer has been exceeded). It'll throw an OutOfMemoryException if it can't get that.

Perhaps that could be the issue? Is the error occurring on the first Replace call or a later one? if it's having to double its size (i believe that's the pattern it uses) multiple times, i could see it running out of contiguous blocks of memory in its address space... maybe...

Ayende Rahien
01/05/2011 03:35 PM by
Ayende Rahien

Tim,

That is likely, except that the size isn't that big, 200 KB is small, and even running in 32 bits, the app only uses about 600 MB, so even if it needed to allocate a full 1MB, I find it hard to believe it couldn't.

Erik Juhlin
01/05/2011 03:42 PM by
Erik Juhlin

I don't think it would solve your problem, but looking at the code it seems to do a lot of Regex replaces. Using RegexOptions.Compiled could really improve performance.

Slower the first time, but then a lot faster. The static RegEx.Replace method will handle the caching of it...

Jesse Houwing
01/05/2011 06:26 PM by
Jesse Houwing

First of all, I think there is an error in your expression, at least I guess there is. This error is very unlikely to be the cause of the exception you're seeing but still...

[\d|] is the equivalent of either a number, or '|' or '', I expect you want: [\d] instead or (\d|).

Second, your Regex.Replace call will replace all instances from the beginning on each pass, which makes it pretty expensive I gather. Have you tried constructing one Regex that searches for all parameter names @(param1|param2|param3|... etc) and using a MatchEvaluator to look up the replacement values? That would allow you to replace all the parameters in one pass... Though it would make the actual expression a lot more expensive...

Both things won't explain the error you're seeing as I have no clue why exactly you're getting it :).

configurator
01/06/2011 12:52 AM by
configurator

Add right before the foreach:

if (statement.Length > 100000)

return "WTF";

fixed?

Comments have been closed on this topic.