http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Ayende, have you tried to compare memory usage of ESENT vs Munin?http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment13http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment13Fri, 07 Dec 2012 03:46:58 GMTYou should create several dumps while the memory is growing to see a pattern in one of the threads to see where it is actually allocating something. Only by looking at the threads while they are not allocating will not lead the way. With VMMap from SysInternals you can have a look at the contents and newly allocated stuff as well. That can also give some clue what the stuff taking so much space actually is. http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment12http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment12Thu, 06 Dec 2012 23:24:36 GMTNote that using memprofiler with unmanaged resources tracker enabled to profile your process will give you more information than using it to analyze a dump file.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment11http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment11Thu, 06 Dec 2012 19:58:16 GMTAyende, I had to mention that memprofiler partially analyses non-managed code. See http://memprofiler.com/OnlineDocs, "Native Memory Page" section. There is no guarantee that it will solve your particular issue, but it is free to try. As for the book, it doesn't cover memprofiler in any detail, but provides several good strategies for memory profiling in general. Not sure they will be new for you, though.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment10http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment10Thu, 06 Dec 2012 19:30:13 GMTA couple of tools I've found helpful fast/easy/simple to use for memory investigation: - CLRProfiler4 (despite having a name that sounds more CPU-centric) from the CLR team (IIRC) - if it has any managed code, mainly because it's so simple and quick to use that it's useful even if only as a verification that the managed heap(s) are fine. Its heap map is very useful, especially in comparison to windbg/sos :) http://www.microsoft.com/en-us/download/details.aspx?id=16273 - Process Explorer - not as detailed as other tools, but its process properties dialog includes a simple way to see all memory info ('Performance' tab) and managed memory ('.NET Performance' tab, especially with '.NET CLR Memory', the default, selected under '.NET Performance Objects') Much like the GitHub's situation with "the bridge loop problem that wasn't there" (https://github.com/blog/1346-network-problems-last-friday), I find it important to use the coarse-granularity tools available to help narrow down the problem (even if it's just to verify a 'hunch') before bringing out the tools like windbg :) Of course, that's exactly what Ayende did here, of course, just reiterating that it's a great idea. :)http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment9http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment9Thu, 06 Dec 2012 19:28:33 GMTChris, I am pretty sure it is an Esent issue, but I am also pretty sure that it isn't the cache. We added some diagnostics, and the cache takes ~400MB, so that isn't counting for a lot of the data, still investigating.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment8http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment8Thu, 06 Dec 2012 19:01:55 GMTRoman, I am pretty sure that the problem is not in managed code, so memprofiler isn't likely to help me here.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment7http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment7Thu, 06 Dec 2012 18:59:33 GMTMy immediate thought when I saw this was Esent too. Setting the cache parameters solved a similar problem for me. I believe there are actually some config options for this in RavenDB, but you probably know that ;)http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment6http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment6Thu, 06 Dec 2012 17:36:17 GMTAs your managed heap is very low, have you considered using DebugDiag's LeakTrack to try and catch native allocations that leak?http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment5http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment5Thu, 06 Dec 2012 17:11:27 GMTLoading the dmp and using !address -summary shows 1.15 Gb of committed memory, but as you note the problem does not seem to be in a managed heap since !dumpheap, !eeheap and !heapstat all show memory consumption less than 100 Mb. This seems to indicate most memory is being allocated and used by unmanaged code. You mentioned that you are using ESENT. Did you see another user ran into a problem with Managed ESENT that caused over 1 GB of memory to be consumed? http://managedesent.codeplex.com/discussions/276175http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment4http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment4Thu, 06 Dec 2012 15:50:44 GMTKeep in mind !dumpheap shows the managed heap. It won't show you the unmanaged memory used by the process, and if you're using esent that points more fingers toward the issue being in unmanaged memory.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment3http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment3Thu, 06 Dec 2012 15:03:25 GMTwindbg is not really user friendly. Even helped by Tess's blog here http://blogs.msdn.com/b/tess/archive/tags/memory+issues/default.aspx?PageIndex=1 I can't figure it out.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment2http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment2Thu, 06 Dec 2012 13:37:54 GMThttp://memprofiler.com is a great memory profiler. See also http://stackoverflow.com/questions/399847/net-memory-profiling-tools and http://www.apress.com/9781430244585 for alternatives.http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment1http://ayende.com/160193/debugging-memory-issues-with-ravendb-using-windbg#comment1Thu, 06 Dec 2012 12:26:47 GMT