Ayende @ Rahien

Refunds available at head office

High memory usage with WCF Discovery

Yes, I know that this is basically saying that select is broken, but I am seeing some very strange stuff here. The code in question is this:

for (int i = 0; i < 15; i++)
{
    var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint());
    var findCriteria = new FindCriteria(typeof(IDiscoverableService))
    {
        Duration = TimeSpan.FromSeconds(1)
    };
    discoveryClient.Find(findCriteria);
    discoveryClient.Close();
}

The full repro can be found here.

The problem is, put simply, that before this code, the working set is:  Working Set: 57,640 kb, after this have executed, it is 90,288 kb.

I went over the code with a fine tooth comb, but I don’t really see where all of this memory have gone.

The actual memory reported by GC.GetTotalMemory does go down after the cleanup, so I guess it could be that .NET isn’t releasing the memory back to the OS, but it still worries me somewhat.

I tried it with higher numbers for the loop, and it seems like eventually it settles down on some number and doesn’t grow from there. My main problem is what happens when you start doing async stuff. Let us take a look here:

int count = 150;
var countdown = new CountdownEvent(count);

for (int i = 0; i < count; i++)
{
    var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint());
    var findCriteria = new FindCriteria(typeof(IDiscoverableService))
    {
    };
    discoveryClient.FindProgressChanged += (sender, eventArgs) => {  }; // do nothing
    discoveryClient.FindCompleted += (sender, eventArgs) =>
    {
        countdown.AddCount();
        discoveryClient.Close();
        PrintMemory("Complete: ");
    };
    discoveryClient.FindAsync(findCriteria);
}

countdown.Wait();

At peek, I am seeing 600 MB (!) of memory used. Note that we are now using the default duration of 20 seconds, and it seems that DiscoveryClient is very heavy on memory.

If you know how, can you take a look at the code and tell me that I am crazy?

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Patric
07/22/2011 11:52 AM by
Patric

Fore some time ago I have a problem with objects that was using events and the original object (discoveryClient) cant be released and disposed. The problem this time was that the object was only referenced by event-handler and because of that you have a reference to the object and the GC was not able to discard the object. After remove the event-handler after the object was used fixed my memory-leak problem.

Ayende Rahien
07/22/2011 02:28 PM by
Ayende Rahien

Patric, But I am not holding any reference to the DiscoveryClient in the event handlers, or anywhere else.

Vadim
07/22/2011 03:38 PM by
Vadim

As an MS intern, I have an MS Q&A tech support card. Tell me if you need one :)

Jon
07/22/2011 04:06 PM by
Jon

Funny enough I was looking at this problem last week about wcf service holding onto large amounts of memory it turned out to be a simple configuration change to the setting maxBufferPoolSize. When you look at a lot of examples about sending large files etc through wcf services everyone blindly sets all the settings (maxBufferSize, maxReceivedMessageSize etc) to huge values and ussually the maxBufferPoolSize get set to the same huge value. However, this value indicates the pool size not a buffer size. so you want to limit your buffer pool to a number of cached buffers and not let it run wild. Set the maxBufferPoolSize to 1 just to test this theory and your memory usage should drop dramtically (if it is this!).

I posted something on stackoverflow about this http://stackoverflow.com/questions/6713180/addressalreadyinuseexception-when-using-custombinding-but-not-when-using-nettcpbi/6720923#6720923

hope this helps!

Ayende Rahien
07/22/2011 05:21 PM by
Ayende Rahien

Jon, Interesting observation, and I tried playing with those values, but I don't think that this is it, I am seeing the same numbers, pretty much, even after limiting the max pool size to just 512 (bytes!).

Ayende Rahien
07/22/2011 05:22 PM by
Ayende Rahien

Vadim, Thanks, I wouldn't want you to "waste" this card unless you really don't have anything else to do with it.

Saj
07/22/2011 05:22 PM by
Saj

I could be wrong but could you try using ( var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint()) ) { var findCriteria = new FindCriteria(typeof(IDiscoverableService)) { Duration = TimeSpan.FromSeconds(1) }; discoveryClient.Find(findCriteria); discoveryClient.Close(); }

Ayende Rahien
07/22/2011 05:24 PM by
Ayende Rahien

Saj, Nope, I am calling Close there.

Remco
07/22/2011 06:21 PM by
Remco

How about bringing out the big guns: procdump from sysinternals to make a dump, and windbg with the psscor4 extensions to show you where the memory has gone. ( with !dumpheap -stat )

Ayende Rahien
07/22/2011 06:23 PM by
Ayende Rahien

Remco, I have done that with dotTrace, I can see that most of the memory is byte[], held by WCF, doesn't really help me figure out what is going on.

Remco
07/22/2011 06:34 PM by
Remco

Really strange, the second example levels out at 210MB on my box (Vista 64bit), with no config of WCF whatsoever.

Ayende Rahien
07/22/2011 06:38 PM by
Ayende Rahien

Remco, Even so, 200MB still seems to be very excessive amount, no?

Kerry
07/22/2011 07:20 PM by
Kerry

This may be simplistic, but is the "discoveryClient.Find(findCriteria)" method disposing of its results after returning?

Ayende Rahien
07/22/2011 07:23 PM by
Ayende Rahien

Kerry, I would hope so, I am disposing on it directly afterward, at any rate.

Remco
07/22/2011 07:30 PM by
Remco

Maybe it's a debugger interaction. running the sample outside of the debugger got me these results with a debugger attached i got 209 for all the results.

Before collect:209,000 After collect:40,000 After finalizers:40,000

so select isn't broken. the debugger is?

full code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.ServiceModel.Discovery;
using System.ServiceModel;

namespace ConsoleApplication1
{
    class Program
    { 
        static void Main(string[] args)
        {
            int count = 150;
            var countdown = new CountdownEvent(count);

            for (int i = 0; i < count; i++)
            {
                var discoveryClient = new DiscoveryClient(new UdpDiscoveryEndpoint()); 
                var findCriteria = new FindCriteria(typeof(IDiscoverableService))
                {
                };
                discoveryClient.FindProgressChanged += (sender, eventArgs) => { }; // do nothing
                discoveryClient.FindCompleted += (sender, eventArgs) =>
                {
                    using (discoveryClient)
                        discoveryClient.Close();

                    PrintMemory(countdown.CurrentCount.ToString());
                    countdown.Signal();
                };
                discoveryClient.FindAsync(findCriteria);
            }

            countdown.Wait();

            PrintMemory("Before collect");
            GC.Collect();
            PrintMemory("After collect");
            GC.WaitForPendingFinalizers();
            PrintMemory("After finalizers");
            Console.ReadKey(); 
        }

        private static void PrintMemory(string p)
        {
            Console.WriteLine("{0}:{1:N3}", p, Environment.WorkingSet / (1024 * 1024));
        }
    }
    [ServiceContract]
    public interface IDiscoverableService
    {
        [OperationContract]
        void dosomething();
    }
}
Remco
07/22/2011 07:49 PM by
Remco

It gets weirder, the results aren't consistent.

only if I add this bit of code before the console.readkey I get a consistent drop of memory usage to about 40 MB

        Thread.Sleep(60000);
        PrintMemory("After sleep");
        GC.Collect();
        PrintMemory("After collect 2 ");
        GC.WaitForPendingFinalizers();
        PrintMemory("After finalizers 2");
        Console.ReadKey();

So my guess there's something fishy going on in the unmanaged world.

Remco
07/22/2011 08:05 PM by
Remco

Last spam of the day, just edited the full repro to include an extra sleep of 60 seconds. Verdict: Select isn't broken, it's garbage collector is slow ;-)

Ayende Rahien
07/22/2011 08:08 PM by
Ayende Rahien

Remco, My problem isn't so much that this is a memory leak, my problem is that it is using so much memory.

Remco
07/22/2011 08:49 PM by
Remco

Ah yes, well, can't help you there. apparently 640K is not enough for even 1 discoveryclient.

Maybe it behaves better memorywise if just 1 discovery client with multiple async discoveries in flight is used instead of a whole bunch of clients? I am unfamiliar with System.ServiceModel.Discovery so I'm just guessing here.

Remco
07/22/2011 09:59 PM by
Remco

It's probably the Binding buffer configuration.

Using the below UdpDiscoveryEndpoint configuration for both service and client there is substantially less memory usage (from 170MB to 67MB maximum working set). Also, discovery is slower.

        var endpoint = new UdpDiscoveryEndpoint();
        var binding = (CustomBinding)endpoint.Binding;
        var element = binding.Elements.OfType<TransportBindingElement>().First();
        element.MaxBufferPoolSize = ushort.MaxValue;
        element.MaxReceivedMessageSize = ushort.MaxValue / 4;
        return endpoint;
Ayende Rahien
07/23/2011 02:23 AM by
Ayende Rahien

Remco, I am probably going to try that next, yes. But I think that the actual cost is per connected server, need to check that further

Ayende Rahien
07/23/2011 02:25 AM by
Ayende Rahien

That is very interesting, because it looks like most of the memory cost is actually on the Service Host, not on the DiscoverClient.

Bertrand Le Roy
07/23/2011 05:59 AM by
Bertrand Le Roy

Just FYI, the post looks all funny in Google Reader, apparently doubly-encoded.

Robert Slaney
07/24/2011 11:11 PM by
Robert Slaney

Just looked through the System.ServiceModel.Discovery assembly using reflector and the Find method on DiscoveryClient instantiates a SyncOperationState object which internally contains ManualResetEvent.

SyncOperationState class is internal but not disposable, and I cannot see anywhere that the ManualResetEvent is closed or disposed.

I suppose the GC will eventually get around to collecting this but would definitely contribute to the excessive memory

Ayende Rahien
07/25/2011 03:19 PM by
Ayende Rahien

Robert, I find it highly unlikely that a single ManualResetEvent would be that expensive.

Comments have been closed on this topic.