A user reported something really mysterious, RavenDB was using a lot of memory on their machine. Why the mystery? Because RavenDB is very careful to account for the memory it uses, exactly because we run into many such cases before.
According to RavenDB’s own stats, we got:
- 300 MB in managed memory
- 300 MB in unmanaged memory
- 200 MB in mmap files
- 1.7 GB total working set
You might notice the discrepancy in numbers. I’ll admit that we have started debugging everything from the bottom up, and in this case, we used “strace –k” to figure out who is doing the allocations. In particular, we watched for anonymous mmap calls (which is a way to ask the OS for memory) and trace who is making these calls. As it turns out, it was the GC. But we are also checking the GC itself for the amount of managed memory we use, and we get a far smaller number. What is going on?
By default, RavenDB is optimized to use as much of the system resources as it can, to provide low latency and lightning responses. One part of that is the use of the Server GC mode with the RetainVM flag. This means that instead of returning the memory to the operating system, the GC was keeping it around in case it will be needed again. In this case, we got to the point where most of the memory in the process was held by the GC in case we’ll need it again.
The fix was to set the RetainVM flag to false, which meant that the GC Will release memory that is not in used back to the OS much more rapidly.