Ayende @ Rahien

It's a girl

Production issue: ASP.Net Cache kills the application

In one of our production deployments, we occasionally get a complete server process crash. Investigating the event log, we have this:

Exception: System.InvalidOperationException

Message: Collection was modified; enumeration operation may not execute.

StackTrace:    at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()

   at System.Web.Hosting.ObjectCacheHost.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.ApplicationManager.TrimCaches(Int32 percent)

   at System.Web.Hosting.CacheManager.CollectInfrequently(Int64 privateBytes)

   at System.Web.Hosting.CacheManager.PBytesMonitorThread(Object state)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

   at System.Threading._TimerCallback.PerformTimerCallback(Object state)

As you can see, this is a case of what appears to be a run of the mill race condition, translated to a process killing exception because it was thrown from a separate thread.

This thread, by the way, is the ASP.Net Cache cleanup thread, and we have no control whatsoever over that. To make things worse, this application doesn’t even use the ASP.NET Cache in any way shape or form.

Any ideas how to resolve this would be very welcome.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Khalid Abuhakmeh
11/16/2012 03:13 PM by
Khalid Abuhakmeh

Could you try disabling the ASP.Net Cache from the web.config or even creating a NoOpCaching provider of your own that does nothing and replacing the default cache provider.

Khalid Abuhakmeh
11/16/2012 03:18 PM by
Khalid Abuhakmeh

Found this little tidbit in the web config. My thinking is that if you don't expire cache or you disableMemoryCollection then that race condition might not show up.

<system.web> </system.web>

Khalid Abuhakmeh
11/16/2012 03:19 PM by
Khalid Abuhakmeh

Sorry, the comment was stripped for some reason.

[caching] [cache disableExpiration="true" disableMemoryCollection="true" />] [/caching]

Wyatt Barnett
11/16/2012 03:21 PM by
Wyatt Barnett

2nding killing the ASP.NET cache dead -- if you need caching use a caching proxy.

To kill the output cache dead, you need to disable the module in the web config. To get midieval you can also remove it from the root web.config in the %SYSTEMROOT%\Microsoft.NET\Framwork\$VERSION\CONFIG\web.config file.

Ayende Rahien
11/16/2012 03:25 PM by
Ayende Rahien

Wyatt, This isn't output caching, this is the CacheManager getting nasty.

Rémi BOURGAREL
11/16/2012 03:34 PM by
Rémi BOURGAREL

By default asp.net use the cache for every scriptressource, are you using any ?

Petar Repac
11/16/2012 03:39 PM by
Petar Repac

From Reflector: internal long TrimCache(int percent) { long num = 0L; Dictionary<MemoryCache, MemoryCacheInfo>.KeyCollection keys = null; lock (this.lock) { if ((this.cacheInfos != null) && (this.cacheInfos.Count > 0)) { keys = this.cacheInfos.Keys; } } if (keys != null) { foreach (MemoryCache cache in keys) { num += cache.Trim(percent); } } return num; }

"cacheInfos" is protected by the lock, but then "keys" points to "cacheInfos" and is not protected by the lock, so Dictionary can change ant any time.

Rémi BOURGAREL
11/16/2012 03:40 PM by
Rémi BOURGAREL

you can also override string GetOutputCacheProviderName(HttpContext context)

In your httpapplication for loggin wich request try to acces to the asp.net cache.

Khalid Abuhakmeh
11/16/2012 04:31 PM by
Khalid Abuhakmeh

Did you try the configuration settings?

If they didn't you can also do a little reflection trick that looks like it would work.

get the CacheManager using reflection and alter the _timer private property. It is currently using this below.

this.timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this.currentPollInterval, this._currentPollInterval);

You could just modify it to be infinite.

this.timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this.currentPollInterval, int.MaxValue);

not saying you won't have the issue pop up again, but it will happen far into the future.

Poul Foged
11/16/2012 04:38 PM by
Poul Foged

Something must be in the cache, try taking a look at Cache Items

Ayende Rahien
11/16/2012 04:40 PM by
Ayende Rahien

Poul, This happens in prod. And I checked, the entire code base doesn't use the cache.

Ayende Rahien
11/16/2012 04:41 PM by
Ayende Rahien

Khalid, That is my nuclear option, I would REALLY not do that.

Poul Foged
11/16/2012 04:48 PM by
Poul Foged

I know, but it can't be empty right? I'd throw up a .aspx that just prints out Cache.Items ....

Ayende Rahien
11/16/2012 04:51 PM by
Ayende Rahien

Poul, * when this happens, it kills the process, no way to get it. * there is no aspx there, it is just a handler that gets all requests.

Poul Foged
11/16/2012 04:57 PM by
Poul Foged

Ok, a handler that spits out cache.items then. So you're 100% sure its empty before the crash? It just feels like a lot of the suggestions here are hacks ...

Ayende Rahien
11/16/2012 04:59 PM by
Ayende Rahien

Poul, I went over the code, there is nothing that touches the cache, yes. And this error happened twice in two weeks period in a heavily occupied prod system.

Khalid Abuhakmeh
11/16/2012 05:01 PM by
Khalid Abuhakmeh

Drastic times call for Drastic measures

Chris Shaffer
11/16/2012 06:10 PM by
Chris Shaffer

Stab in the dark for a fix and not an explanation - Maybe put something in the cache every once in a while so the cleanup thread has something to do.

Kevin Dente
11/16/2012 06:25 PM by
Kevin Dente

It's not a fix, but if you want to stabilize things temporarily you could try reverting to the old unhandled exception policy using the legacyUnhandledExceptionPolicy flag in web.config

http://msdn.microsoft.com/en-us/library/ms228965.aspx

Ayende Rahien
11/16/2012 08:04 PM by
Ayende Rahien

Kevin, Thanks, that is what we might do

Ayende Rahien
11/16/2012 08:05 PM by
Ayende Rahien

Mystical, Does it matter if you call each other or it is a loop? It is the same basic principal.

Steven Casey
11/16/2012 08:23 PM by
Steven Casey

did you try disablememorycollection as Khalid suggested?

stackoverflow also suggests the same http://stackoverflow.com/questions/6843343/disable-asp-net-cache

Chris Shaffer
11/16/2012 08:38 PM by
Chris Shaffer

Using DisableMemoryCollection seems like a bad/dangerous idea - then if some bit of the code actually is using it (whether that bit of code is something built into .Net, a 3rd party library, or the code), then you risk wasting memory and the out of memory issue mentioned in the linked answer. And you just have to remember to be careful with the cache in the future (as does anyone that happens to inherit the project in the future).

Kevin Dente's suggestion seems the most appropriate as a fix for the symptom - perhaps add on a memory dump when the exception occurs so you can try to get down to the root cause later on...

Remco Ros
11/18/2012 06:42 PM by
Remco Ros

Pretty awesome that you can post questions like this, and get responses faster then when we post a question on stackoverflow.. epic, and well deserved ofcourse.

Andrew
11/18/2012 10:13 PM by
Andrew

Is it possible you have another website in the same app pool which is responsible for the crash?

Rory Primrose
11/18/2012 11:02 PM by
Rory Primrose

You might not be putting something in the cache in your code, but you may be using another component that does.

Ayende Rahien
11/19/2012 08:15 AM by
Ayende Rahien

Andrew, No, there is only one site on this machine.

Francisco Gómez
11/19/2012 03:32 PM by
Francisco Gómez

Perhaps the exception of the log is not the cause but a restart of the application for memory problems (App Domain recycling). Enable tracing "Application Lifetime Events" to see what´s going on before the exception.

smcintire
11/20/2012 01:32 PM by
smcintire

Are you using iis 7.5? if so, is the application running in integrated or classic pipe (and does it happen in both)? which version of .net are you running in? can you change the max cache size in your iis config?

Kasi Prasad
12/01/2012 01:10 AM by
Kasi Prasad

Sounds like a job for DebugDiag.

You might just try:

<system.webServer> </system.webServer>

Kasi Prasad
12/01/2012 01:12 AM by
Kasi Prasad

My comment got clipped:

<system.webServer> < caching enabled="false" /> </system.webServer>

Comments have been closed on this topic.