+972 52-548-6969
Production issue: ASP.Net Cache kills the application
In one of our production deployments, we occasionally get a complete server process crash. Investigating the event log, we have this:
Exception: System.InvalidOperationException
Message: Collection was modified; enumeration operation may not execute.
StackTrace: at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()
at System.Web.Hosting.ObjectCacheHost.TrimCache(Int32 percent)
at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)
at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)
at System.Web.Hosting.ApplicationManager.TrimCaches(Int32 percent)
at System.Web.Hosting.CacheManager.CollectInfrequently(Int64 privateBytes)
at System.Web.Hosting.CacheManager.PBytesMonitorThread(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading._TimerCallback.PerformTimerCallback(Object state)
As you can see, this is a case of what appears to be a run of the mill race condition, translated to a process killing exception because it was thrown from a separate thread.
This thread, by the way, is the ASP.Net Cache cleanup thread, and we have no control whatsoever over that. To make things worse, this application doesn’t even use the ASP.NET Cache in any way shape or form.
Any ideas how to resolve this would be very welcome.
Comments
Could you try disabling the ASP.Net Cache from the web.config or even creating a NoOpCaching provider of your own that does nothing and replacing the default cache provider.
How do you do that?
Found this little tidbit in the web config. My thinking is that if you don't expire cache or you disableMemoryCollection then that race condition might not show up.
<system.web> <caching> <cache disableExpiration="true" disableMemoryCollection="true" /> </caching> </system.web>
Sorry, the comment was stripped for some reason.
[caching] [cache disableExpiration="true" disableMemoryCollection="true" />] [/caching]
2nding killing the ASP.NET cache dead -- if you need caching use a caching proxy.
To kill the output cache dead, you need to disable the module in the web config. To get midieval you can also remove it from the root web.config in the %SYSTEMROOT%\Microsoft.NET\Framwork\$VERSION\CONFIG\web.config file.
Wyatt, This isn't output caching, this is the CacheManager getting nasty.
By default asp.net use the cache for every scriptressource, are you using any ?
From Reflector: internal long TrimCache(int percent) { long num = 0L; Dictionary<MemoryCache, MemoryCacheInfo>.KeyCollection keys = null; lock (this._lock) { if ((this._cacheInfos != null) && (this._cacheInfos.Count > 0)) { keys = this._cacheInfos.Keys; } } if (keys != null) { foreach (MemoryCache cache in keys) { num += cache.Trim(percent); } } return num; }
"_cacheInfos" is protected by the lock, but then "keys" points to "_cacheInfos" and is not protected by the lock, so Dictionary can change ant any time.
you can also override string GetOutputCacheProviderName(HttpContext context)
In your httpapplication for loggin wich request try to acces to the asp.net cache.
Remi, No, I do not.
Did you try the configuration settings?
If they didn't you can also do a little reflection trick that looks like it would work.
get the CacheManager using reflection and alter the _timer private property. It is currently using this below.
this._timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this._currentPollInterval, this._currentPollInterval);
You could just modify it to be infinite.
this._timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this._currentPollInterval, int.MaxValue);
not saying you won't have the issue pop up again, but it will happen far into the future.
Something must be in the cache, try taking a look at Cache Items
Poul, This happens in prod. And I checked, the entire code base doesn't use the cache.
Khalid, That is my nuclear option, I would REALLY not do that.
I know, but it can't be empty right? I'd throw up a .aspx that just prints out Cache.Items ....
Poul, * when this happens, it kills the process, no way to get it. * there is no aspx there, it is just a handler that gets all requests.
Ok, a handler that spits out cache.items then. So you're 100% sure its empty before the crash? It just feels like a lot of the suggestions here are hacks ...
Poul, I went over the code, there is nothing that touches the cache, yes. And this error happened twice in two weeks period in a heavily occupied prod system.
Drastic times call for Drastic measures
Stab in the dark for a fix and not an explanation - Maybe put something in the cache every once in a while so the cleanup thread has something to do.
It's not a fix, but if you want to stabilize things temporarily you could try reverting to the old unhandled exception policy using the legacyUnhandledExceptionPolicy flag in web.config
http://msdn.microsoft.com/en-us/library/ms228965.aspx
Kevin, Thanks, that is what we might do
Mystical, Does it matter if you call each other or it is a loop? It is the same basic principal.
did you try disablememorycollection as Khalid suggested?
stackoverflow also suggests the same http://stackoverflow.com/questions/6843343/disable-asp-net-cache
Using DisableMemoryCollection seems like a bad/dangerous idea - then if some bit of the code actually is using it (whether that bit of code is something built into .Net, a 3rd party library, or the code), then you risk wasting memory and the out of memory issue mentioned in the linked answer. And you just have to remember to be careful with the cache in the future (as does anyone that happens to inherit the project in the future).
Kevin Dente's suggestion seems the most appropriate as a fix for the symptom - perhaps add on a memory dump when the exception occurs so you can try to get down to the root cause later on...
Pretty awesome that you can post questions like this, and get responses faster then when we post a question on stackoverflow.. epic, and well deserved ofcourse.
Is it possible you have another website in the same app pool which is responsible for the crash?
You might not be putting something in the cache in your code, but you may be using another component that does.
Andrew, No, there is only one site on this machine.
Perhaps the exception of the log is not the cause but a restart of the application for memory problems (App Domain recycling). Enable tracing "Application Lifetime Events" to see what´s going on before the exception.
Are you using iis 7.5? if so, is the application running in integrated or classic pipe (and does it happen in both)? which version of .net are you running in? can you change the max cache size in your iis config?
Sounds like a job for DebugDiag.
You might just try:
<system.webServer> <caching enabled="false" /> </system.webServer>
My comment got clipped:
<system.webServer> < caching enabled="false" /> </system.webServer>
Comment preview