Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 18 | Comments: 86

filter by tags archive

ChallengeYour own ThreadLocal

time to read 4 min | 767 words

I found a bug in Lucene.NET that resulted in a memory leak in RavenDB. Take a look here (I have simplified the code to some extent, but the same spirit remains):

public class CloseableThreadLocal
{
    [ThreadStatic] private static Dictionary<object, object> slots;

    public static Dictionary<object, object> Slots
    {
        get { return slots ?? (slots = new Dictionary<object, object>()); }
    }

    public /*protected internal*/ virtual Object InitialValue()
    {
        return null;
    }

    public virtual Object Get()
    {
        object val;

        if (Slots.TryGetValue(this, out val))
        {
            return val;
        }
        val = InitialValue();
        Set(val);
        return val;
    }

    public virtual void Set(object val)
    {
        Slots[this] = val;
    }

    public virtual void Close()
    {
        if (slots != null)// intentionally using the field here, to avoid creating the instance
            slots.Remove(this);
    }
}

As you can imagine, this is a fairly elegant way of doing this (please note, .NET 4.0 have the ThreadLocal class, which I strongly recommend using). But it has one very serious flaw. It you don’t close the instance, you are going to leak some memory. As you can imagine, that is a pretty bad thing to do.

In general, I consider such designs as pretty bad bugs when writing managed code. We have the GC for a reason, and writing code that forces the user to manually manage memory is BAD for you. Here is an example showing the problem:

class Program
{
    static void Main(string[] args)
    {
        UseThreadLocal();
        GC.Collect(2);
GC.WaitForPendingFinalizers();
Console.WriteLine(CloseableThreadLocal.slots.Count); } private static void UseThreadLocal() { var tl = new CloseableThreadLocal(); tl.Set("hello there"); Console.WriteLine(tl.Get()); } }

This will show that after the UseThreadLocal() run and we force full collection, the value is still there.

Without using the builtin ThreadLocal, can you figure out a way to solve this?

Points goes to whoever does this with the minimum amount of changes to the code.

More posts in "Challenge" series:

  1. (28 Apr 2015) What is the meaning of this change?
  2. (26 Sep 2013) Spot the bug
  3. (27 May 2013) The problem of locking down tasks…
  4. (17 Oct 2011) Minimum number of round trips
  5. (23 Aug 2011) Recent Comments with Future Posts
  6. (02 Aug 2011) Modifying execution approaches
  7. (29 Apr 2011) Stop the leaks
  8. (23 Dec 2010) This code should never hit production
  9. (17 Dec 2010) Your own ThreadLocal
  10. (03 Dec 2010) Querying relative information with RavenDB
  11. (29 Jun 2010) Find the bug
  12. (23 Jun 2010) Dynamically dynamic
  13. (28 Apr 2010) What killed the application?
  14. (19 Mar 2010) What does this code do?
  15. (04 Mar 2010) Robust enumeration over external code
  16. (16 Feb 2010) Premature optimization, and all of that…
  17. (12 Feb 2010) Efficient querying
  18. (10 Feb 2010) Find the resource leak
  19. (21 Oct 2009) Can you spot the bug?
  20. (18 Oct 2009) Why is this wrong?
  21. (17 Oct 2009) Write the check in comment
  22. (15 Sep 2009) NH Prof Exporting Reports
  23. (02 Sep 2009) The lazy loaded inheritance many to one association OR/M conundrum
  24. (01 Sep 2009) Why isn’t select broken?
  25. (06 Aug 2009) Find the bug fixes
  26. (26 May 2009) Find the bug
  27. (14 May 2009) multi threaded test failure
  28. (11 May 2009) The regex that doesn’t match
  29. (24 Mar 2009) probability based selection
  30. (13 Mar 2009) C# Rewriting
  31. (18 Feb 2009) write a self extracting program
  32. (04 Sep 2008) Don't stop with the first DSL abstraction
  33. (02 Aug 2008) What is the problem?
  34. (28 Jul 2008) What does this code do?
  35. (26 Jul 2008) Find the bug fix
  36. (05 Jul 2008) Find the deadlock
  37. (03 Jul 2008) Find the bug
  38. (02 Jul 2008) What is wrong with this code
  39. (05 Jun 2008) why did the tests fail?
  40. (27 May 2008) Striving for better syntax
  41. (13 Apr 2008) calling generics without the generic type
  42. (12 Apr 2008) The directory tree
  43. (24 Mar 2008) Find the version
  44. (21 Jan 2008) Strongly typing weakly typed code
  45. (28 Jun 2007) Windsor Null Object Dependency Facility

Comments

tobi

Implement IDisposable for the common case and Finalize() for the uncommon/bug case. The finalizer could throw if a debugger was attached to assist in finding leaks.

Steve Py

Well, my guess based on the behaviour you describe is to make the class disposable. On dispose, call Close. Anyone implementing the class should be checking for IDisposable and be "using" appropriately.

Other than that it may be a candidate scenario for calling Close() within a finalizer, though that test scenario won't pass with the finalizer implementation likely due to compiler optimization, but I believe it would prevent the memory leak.

I'd opt for best practice though.

Rob Ashton

Thinking about this now,

IDisposable doesn't solve anything, it just gives the class a pattern that people recognise so they're more likely to remember to call Close.

Using a finalizer is a great idea, except you've got a ThreadStatic there, and you don't know what thread the finalizer will be called in.

(So calling Close from a finalizer will not work)

Okay, so I've given my opinion on why I think those won't work, I'm still thinking about the best solution =)

Rafal

But how does the memory leak occur? [ThreadLocal] objects should be GCd after the thread terminates, so the only possibility is that the thread doesn't terminate and is reused for other purposes, like in a thread pool.

tobi

Ok, finalization happens on a different thread... thats nasty! Looks like i am still not done with learning.

You could first convert the dictionary into a hashset and store the value in the instance. Then you change the hashset to a weakhashset (not built-in). then you introduce a threadstatic int counter which you use to purge the dead items on every 1000th operation. what a hack...

depending on the perf requirements i would just use a localdatastoreslot which is built-in on 2.0 upwards.

Thomas Eyde

Using WeakReference is the first thing that pops into my mind

Steve Py

Rob's right, Cannot rely on a finalizer. Frankly I'd choose the big stick approach. Implement IDisposable, and a finalizer that throws if Dispose wasn't called.

private bool _isDisposed = false;

void IDisposable.Dispose()

{

Dispose(true);

}

protected virtual void Dispose(bool disposing)

{

if (!_isDisposed)

{

    if (disposing)

        Close();

       _isDisposed = true;

}

}

~CloseableThreadLocal()

{

if (!_isDisposed)

{

if DEBUG

    Debugger.Break();

endif

    throw new InvalidOperationException("Disposable object was not disposed. (CloseableThreadLocal)");

}

}

It is a memory leak because on a long-lived thread repeated use of new ThreadLocal instances will continue to pile up items in the static dictionary. If Close isn't called those items are never removed.

tobi

And I just peeked under the covers on .net 4. the implementation is really creative and quite funny ;-) I am very keen to find out about how to do this with "minimal changes".

Here is a potential way: Let all access to the dictionary pass through a threadlocal spinlock. When creating the CloseableThreadLocal in the ctor, copy the dict and the lock to instance variables so that they are available on the finalizer. On the finalizer call close. I understand locks are very cheap if they are nearly always uncontended and thread-local (which would be the case here).

Rob Ashton

The best solution I can come up with WeakReference (I'm at work so I'm having to do this on idle ;-)

    public class CloseableThreadLocal

    {

        [ThreadStatic]

        private static Dictionary

<object,> slots;

        private WeakReference self;


        public CloseableThreadLocal() { self = new WeakReference(this); }


        public static Dictionary

<object,> Slots

        {

            get { return slots ?? (slots = new Dictionary

<object,> ()); }

        }


        public /*protected internal*/ virtual Object InitialValue()

        {

            return null;

        }


        public virtual Object Get()

        {

            object val;


            if (Slots.TryGetValue(self, out val))

            {

                return val;

            }

            val = InitialValue();

            Set(val);

            return val;

        }


        public virtual void Set(object val)

        {

            Slots[self] = val;

        }


        public virtual void Close()

        {

            if (slots != null)// intentionally using the field here, to avoid creating the instance

                slots.Remove(self);

        }

    }

That gives us the ability to add another method to clear anything in the dictionary in the current thread when an object is created on that thread (It's a bit sweep and cleany). (We can check the dictionary for any weak references that haven't got a valid reference anymore)

Seems a bit overkill though given we're going for as few changes as possible, I suspect there is something I'm missing here.

Ryan Heath

There are two problems to overcome:

  • the class is inserting itself into the dictionary so it can never be collected

  • when above problem is fixed, the finalizer is running on a different thread, so we must keep a ref to the slots created in main thread.

public class CloseableThreadLocal

{

[ThreadStatic] private static Dictionary <object,> slots;

object key = new Object();

Dictionary <object,> localSlots = null;

public static Dictionary <object,> Slots

{

get { return slots ?? (slots = new Dictionary

<object,> ()); }

}

Dictionary <object,> LocalSlots

{

get { return slots ?? ( localSlots = Slots); }

}

public /protected internal/ virtual Object InitialValue()

{

return null;

}

public virtual Object Get()

{

object val;


if (LocalSlots.TryGetValue(key, out val))

{

  return val;

}

val = InitialValue();

Set(val);

return val;

}

public virtual void Set(object val)

{

LocalSlots[key] = val;

}

public virtual void Close()

{

if (localSlots != null)// intentionally using the field here, to avoid creating the instance

  localSlots.Remove(key);

}

~CloseableThreadLocal()

{

Close();

}

}

// Ryan

Daniel Grunwald

Here are some additional problems to consider:

  • The Dictionary is keeping the key object alive, so a finalizer will never be called (unless the thread exits).

  • The value object may have a reference to the CloseableThreadLocal

(key object). Such a cyclic reference shouldn't prevent us from garbage collecting the object.

The first problem alone could be worked around by using a Dictionary <weakreference,> and adding a finalizer. But I don't see any way to work around the second problem using .NET 3.5. There doesn't seem to be any way to distinguish between outstanding references from GC roots and outstanding references from our cycles.

In .NET 4.0, it's easy: just replace the Dictionary <object,object> with a ConditionalWeakTable <object,object> .

Then again, the .NET 4.0 ThreadLocal class also doesn't seem to solve the problem with cyclic references; so I guess a Dictionary <weakreference,> is good enough for most usecases.

But I like how the .NET 4.0 implementation manages to directly use the fast [ThreadStatic] support without having to go through slow hash tables.

Daniel Grunwald

This comment box seems to be eating generics...

I meant using a Dictionary[of WeakReference, object] and finalizer.

Rob and Ryan posted in the meantime, but I think both their approaches are incorrect:

Rob is missing the finalizer; and Ryan is clearing only one dictionary - but I would expect that dropping the reference to the CloseableThreadLocal gets rid of the values on ALL threads (I would expect the same of the Close() method, so this issue is already present in Ayende's code).

But I like how Ryan avoided the WeakReference by simply using another object as key.

So instead of the "Dictionary localSlots" (race condition!), I would use a "List[of Dictionary] allDicts" to store the dictionaries of all threads on which this CloseableThreadLocal was used. Except that opens up the possibility of referencing the dictionaries of threads that exited, so you might have to use something like a list of weak references to the dictionaries.

But I don't see how anything except ConditionalWeakTable can stop this simple class from leaking:

class Leak {

CloseableThreadLocal local;

public Leak() {

   local = new CloseableThreadLocal();

   local.Set(this);

}

}

Patrick Huizinga

Daniel, I'm pretty sure WeakReference can deal with cyclic references, since the GC can handle them too.

The GC will determine the only way to reach your Leak instance is through a WeakReference, which it should disregard, and therefore clean up your instance.

Daniel Grunwald

Patrick Huizinga: But there's the following strong reference chain:

ThreadStatic -> slots -> Values -> Leak -> CloseableThreadLocal

Therefore, the finalizer of CloseableThreadLocal doesn't get invoked, and the entry is never removed from the slots.

Yes, WeakReference on its own can handle cycles; it's the static dictionary that's introducing the trouble here. And you can't solve the problem by introducing more WeakReferences. What you need is a CONDITIONAL weak reference: it's a strong reference as long as the key is alive; but weak otherwise.

In GC literature, this kind of reference is called "Ephemerons" ( http://en.wikipedia.org/wiki/Ephemeron).

Ephemeron support is new in the .NET 4.0 garbage collector, and ConditionalWeakTable is the public API for it.

jonnii

This is my answer.

public class CloseableThreadLocal

{

    [ThreadStatic]

    private static Dictionary

<object,> slots;

    private readonly object key = new object();

    private Dictionary

<object,> localref;

    public static Dictionary

<object,> Slots

    {

        get

        {

            return slots ?? (slots = new Dictionary

<object,> ());

        }

    }


    public /*protected internal*/ virtual Object InitialValue()

    {

        return null;

    }


    public Dictionary

<object,> LocalSlots

    {

        get { return localref = Slots; }

    }


    public virtual Object Get()

    {

        object val;


        if (LocalSlots.TryGetValue(key, out val))

        {

            return val;

        }

        val = InitialValue();

        Set(val);

        return val;

    }


    public virtual void Set(object val)

    {

        LocalSlots[key] = val;

    }


    public virtual void Close()

    {

        if (localref != null)

        {

            // intentionally using the field here, to avoid creating the instance

            localref.Remove(key);

        }

    }


    ~CloseableThreadLocal()

    {

        Close();

    }

}

Use an object as the key, so that you don't end up with a reference to yourself and also keep a local reference to the dictionary so that you get a chance to remove yourself.

Patrick Huizinga

jonnii,

This will break your code:

var ctl = new CloseableThreadLocal()

new Thread(() => ctl.Set(new BigObject())).Start();

new Thread(() => ctl.Set(new BigObject())).Start();

It will not GC one of the BigObjects. Also ctl will never be GC'ed.

Patrick Huizinga

hmm, made mistake in criticizing jonnii; ctl does get GC'ed.

Anyway, after shooting down jonnii, it's only fair he gets a shot at my attempt :-)

class ClosableThreadLocal

{

[ThreadStatic] static Dictionary slots;

readonly List usedSlots = new List();

public Dictionary Slots

{

get

{

if (slots != null) return slots;

slots = new Dictionary();

lock (usedSlots) usedSlots.Add(slots);

return slots;

}

}

public object Key { get { return usedSlots; } }

public void Set(object value)

{

Slots[Key] = value;

}

public void Close()

{

if (slots == null) return;

slots.Remove(Key);

lock (usedSlots) usedSlots.Remove(slot);

}

void Dispose()

{

foreach(Dictionary slot in usedSlots) slot.Remove(Key);

}

}

As you can see it's mostly the same code as yours (jonnii), except that I keep a list with all used dictionaries, so I can actually enumerate them all and clean them all out.

Ryan Heath

I agree that a list of dictionaries is saver, but it is questionable that one is using an instance of this class in multiple threads at once.

Smells like a bug...

// Ryan

Justin Chase

Don't use ThreadStatic. Create an instance for whatever scope you want this functionality and dispose of it properly when you leave that scope.

Thomas Eyde

I finally got it. And no need for weak references.

Key points, as previously mentioned in this thread:

  1. We need a thread global reference to track each thread local slot so the finalizer can reach it.

  2. We need to remove the strong reference introduced by the dictionary key. I went for a local Guid.

  3. The finalizer must call Close

  4. Handle new potential memory leaks introduced by code changes.

Code and tests here: http://pastebin.com/x26qnpLn

tobi

Thomas, you need to add locking around the global HashSet. Not sure how performant this is compared to just using LocalDataStoreSlot because the locks will give up many performance gains.

Bernhard Hofmann

Why use [ThreadStatic] when you're not using a new thread for each CloseableThreadLocal?

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Buffer allocation strategies: A possible solution - 3 days from now
  2. Buffer allocation strategies: Explaining the solution - 4 days from now
  3. Buffer allocation strategies: Bad usage patterns - 5 days from now
  4. The useless text book algorithms - 6 days from now
  5. Find the bug: The concurrent memory buster - 7 days from now

There are posts all the way to Sep 11, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    03 Sep 2015 - The industry at large
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats