Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:


+972 52-548-6969

, @ Q c

Posts: 6,128 | Comments: 45,550

filter by tags archive

Production issue: ASP.Net Cache kills the application

time to read 1 min | 140 words

In one of our production deployments, we occasionally get a complete server process crash. Investigating the event log, we have this:

Exception: System.InvalidOperationException

Message: Collection was modified; enumeration operation may not execute.

StackTrace:    at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()

   at System.Web.Hosting.ObjectCacheHost.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.ApplicationManager.TrimCaches(Int32 percent)

   at System.Web.Hosting.CacheManager.CollectInfrequently(Int64 privateBytes)

   at System.Web.Hosting.CacheManager.PBytesMonitorThread(Object state)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

   at System.Threading._TimerCallback.PerformTimerCallback(Object state)

As you can see, this is a case of what appears to be a run of the mill race condition, translated to a process killing exception because it was thrown from a separate thread.

This thread, by the way, is the ASP.Net Cache cleanup thread, and we have no control whatsoever over that. To make things worse, this application doesn’t even use the ASP.NET Cache in any way shape or form.

Any ideas how to resolve this would be very welcome.


Khalid Abuhakmeh

Could you try disabling the ASP.Net Cache from the web.config or even creating a NoOpCaching provider of your own that does nothing and replacing the default cache provider.

Khalid Abuhakmeh

Found this little tidbit in the web config. My thinking is that if you don't expire cache or you disableMemoryCollection then that race condition might not show up.

<system.web> </system.web>

Khalid Abuhakmeh

Sorry, the comment was stripped for some reason.

[caching] [cache disableExpiration="true" disableMemoryCollection="true" />] [/caching]

Wyatt Barnett

2nding killing the ASP.NET cache dead -- if you need caching use a caching proxy.

To kill the output cache dead, you need to disable the module in the web config. To get midieval you can also remove it from the root web.config in the %SYSTEMROOT%\Microsoft.NET\Framwork\$VERSION\CONFIG\web.config file.

Ayende Rahien

Wyatt, This isn't output caching, this is the CacheManager getting nasty.


By default asp.net use the cache for every scriptressource, are you using any ?

Petar Repac

From Reflector: internal long TrimCache(int percent) { long num = 0L; Dictionary<MemoryCache, MemoryCacheInfo>.KeyCollection keys = null; lock (this.lock) { if ((this.cacheInfos != null) && (this.cacheInfos.Count > 0)) { keys = this.cacheInfos.Keys; } } if (keys != null) { foreach (MemoryCache cache in keys) { num += cache.Trim(percent); } } return num; }

"cacheInfos" is protected by the lock, but then "keys" points to "cacheInfos" and is not protected by the lock, so Dictionary can change ant any time.


you can also override string GetOutputCacheProviderName(HttpContext context)

In your httpapplication for loggin wich request try to acces to the asp.net cache.

Khalid Abuhakmeh

Did you try the configuration settings?

If they didn't you can also do a little reflection trick that looks like it would work.

get the CacheManager using reflection and alter the _timer private property. It is currently using this below.

this.timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this.currentPollInterval, this._currentPollInterval);

You could just modify it to be infinite.

this.timer = new Timer(new TimerCallback(this.PBytesMonitorThread), (object) null, this.currentPollInterval, int.MaxValue);

not saying you won't have the issue pop up again, but it will happen far into the future.

Poul Foged

Something must be in the cache, try taking a look at Cache Items

Ayende Rahien

Poul, This happens in prod. And I checked, the entire code base doesn't use the cache.

Ayende Rahien

Khalid, That is my nuclear option, I would REALLY not do that.

Poul Foged

I know, but it can't be empty right? I'd throw up a .aspx that just prints out Cache.Items ....

Ayende Rahien

Poul, * when this happens, it kills the process, no way to get it. * there is no aspx there, it is just a handler that gets all requests.

Poul Foged

Ok, a handler that spits out cache.items then. So you're 100% sure its empty before the crash? It just feels like a lot of the suggestions here are hacks ...

Ayende Rahien

Poul, I went over the code, there is nothing that touches the cache, yes. And this error happened twice in two weeks period in a heavily occupied prod system.

Chris Shaffer

Stab in the dark for a fix and not an explanation - Maybe put something in the cache every once in a while so the cleanup thread has something to do.

Kevin Dente

It's not a fix, but if you want to stabilize things temporarily you could try reverting to the old unhandled exception policy using the legacyUnhandledExceptionPolicy flag in web.config


Ayende Rahien

Kevin, Thanks, that is what we might do

Ayende Rahien

Mystical, Does it matter if you call each other or it is a loop? It is the same basic principal.

Steven Casey

did you try disablememorycollection as Khalid suggested?

stackoverflow also suggests the same http://stackoverflow.com/questions/6843343/disable-asp-net-cache

Chris Shaffer

Using DisableMemoryCollection seems like a bad/dangerous idea - then if some bit of the code actually is using it (whether that bit of code is something built into .Net, a 3rd party library, or the code), then you risk wasting memory and the out of memory issue mentioned in the linked answer. And you just have to remember to be careful with the cache in the future (as does anyone that happens to inherit the project in the future).

Kevin Dente's suggestion seems the most appropriate as a fix for the symptom - perhaps add on a memory dump when the exception occurs so you can try to get down to the root cause later on...

Remco Ros

Pretty awesome that you can post questions like this, and get responses faster then when we post a question on stackoverflow.. epic, and well deserved ofcourse.


Is it possible you have another website in the same app pool which is responsible for the crash?

Rory Primrose

You might not be putting something in the cache in your code, but you may be using another component that does.

Ayende Rahien

Andrew, No, there is only one site on this machine.

Francisco Gómez

Perhaps the exception of the log is not the cause but a restart of the application for memory problems (App Domain recycling). Enable tracing "Application Lifetime Events" to see what´s going on before the exception.


Are you using iis 7.5? if so, is the application running in integrated or classic pipe (and does it happen in both)? which version of .net are you running in? can you change the max cache size in your iis config?

Kasi Prasad

Sounds like a job for DebugDiag.

You might just try:

<system.webServer> </system.webServer>

Kasi Prasad

My comment got clipped:

<system.webServer> < caching enabled="false" /> </system.webServer>

Comment preview

Comments have been closed on this topic.


  1. The worker pattern - 16 hours from now

There are posts all the way to May 30, 2016


  1. The design of RavenDB 4.0 (14):
    26 May 2016 - The client side
  2. RavenDB 3.5 whirl wind tour (14):
    25 May 2016 - Got anything to declare, ya smuggler?
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series


Main feed Feed Stats
Comments feed   Comments Feed Stats