During a performance problem investigation, we discovered that the following innocent looking code was killing our performance.

This is part of a code that allow users to subscribe to changes in the database using a WebSocket. This is pretty rare, so we check that there aren’t any and skip all the work.

We had a bunch of code that run on many threads and ended up calling this method. Since there are not subscribers, this should be very cheap, but it wasn’t. The problem was that the call to Count was locking, and that created a convoy that killed our performance.

We did some trawling through our code base and through the framework code and came up with the following conclusions.

ConcurrentQueue:

• Clear locks
• CopyTo, GetEnumerator, ToArray, Count creates a snapshot (consume more memory)
• TryPeek, IsEmpty are cheap

Here is a piece of problematic code, we are trying to keep the last ~25 items that we looked at:

The problem is that this kind of code ensures that there will be a lot of snapshots and increases memory utilization.

ConcurrentDictionary:

• Count, Clear, IsEmpty, ToArray  - lock the entire thing
• TryAdd, TryUpdate, TryRemove – lock the bucket for this entry
• GetEnumerator does not lock
• Keys, Values both lock the table and forces a temporary collection

If you need to iterate over the keys of a concurrent dictionary, there are two options:

Iterating over the entire dictionary is much better then iterating over just the keys.

There have been some changes, and it seems that it is hard to track them. Here are where you can find the master repositories for the rhino tools projects:

• Rhino Commons - https://github.com/lorenzomelato/rhino-commons
• Rhino Service Bus- https://github.com/hibernating-rhinos/rhino-esb
• Rhino Licensing - https://github.com/hibernating-rhinos/rhino-licensing
• Rhino ETL - https://github.com/hibernating-rhinos/rhino-etl
• Rhino Security - https://github.com/hibernating-rhinos/rhino-security
• Rhino Persistent Hash Table - https://github.com/hibernating-rhinos/rhino-pht
• Rhino Distributed Hash Table - https://github.com/hibernating-rhinos/rhino-dht
• Rhino DSL - https://github.com/hibernating-rhinos/rhino-dsl
• Rhino Mocks - https://github.com/hibernating-rhinos/rhino-mocks
• Rhino Queues - https://github.com/hibernating-rhinos/rhino-queues