+972 52-548-6969
ThreadPool vs Pool<Thread>
One of the changes we made to RavenDB 4.0 as part of the production run feedback was to introduce the notion of a pool of threads. This is quite distinct from the notion of a thread pool, and it deserves it own explanation.
A thread pool is something that is very commonly used in server applications. Instead of spawning a thread per task (a really expensive process), the system keeps a pool of threads around and provide some way to queue tasks for them to do. Such tasks are typically expected to be short and transient. They should also not have any expectation about the state of the thread nor should they modify any thread state.
In .NET, all async work will typically go through the system thread pool. When you are processing a request in ASP.Net, you are running on a thread pool thread and it is in heavy use in any sort of server environment.
RavenDB is also making heavy use of the thread pool, to service requests and handle most operations. But it also has the need to process several long terms tasks (which can run for days or more). Because of that, and because we need both fine grained control and the ability to inspect the state of the system easily, we typically spawn a new, dedicated, thread for such tasks. As it turns out, under high memory load, this is a dangerous thing to do. The thread might try to commit some stack space, but there is no memory in the system to do so, resulting in a fatal stack overflow.
I should note that the kind of tasks we use a dedicated thread for are pretty rare and long lived, they also do things like mutate the thread state (changing the priority, for example), for example.
Because of that, we can’t just use the thread pool, nor do we want a similar abstraction. Instead, we created a pool of threads. A job can request a thread to run on, and it will get its own thread to run and do with as it pleases. When it is done running, which can be in a minute or in a week’s time, it will return the thread to the pool, where it will remain until another job needs it.
In this way, under high memory usage, we’ll not be creating new threads all the time, and the threads’ stack are likely to be already committed and available to the process.
Update: To clear things up. Even if we do need to create a new thread, we now have control over that, in a single place. If there isn't enough memory available to actually use the new thread stack, we'll refuse to create it.
Comments
Do indexes still use one thread per index (as detailed in https://ayende.com/blog/179940/ravendb-4-0-unsung-heroes-the-indexing-threads ) or do they use this scheme instead? Many of the things you mention, like long-livedness and explicitly changing thread state seem like things that motivated the promotion to thread in the first place in the original post, but using this way you'd also avoid triggering the fatal stack overflow mentioned in this post. All at the cost of a performance degraded but still available server.
Jesper, No, this is basically replacing anywhere we used
new Threadbefore. And it applies to indexing, yes. Note that this is a dedicated thread for this. It is just going to be reused by us rather than be discarded and a new thread going to be created. There should be no perf changes as a result.Gotcha. I meant that if you had 150 indexes, it wouldn't keep one thread alive per index. It would have a pool of threads, and run the index work on them and they, along with other long-running work, would "rent" a thread from this extra pool of threads for as long as they have anything to run. When I said "performance degraded but still available server", I meant in the sense that in a scenario that would otherwise have completely tanked the system or caused the stack overflow you mentioned, the Raven would keep running albeit a bit congested compared to if more resources had been available - so in this way there would be an infinite positive performance improvement compared to Raven crashing. Is this accurate?
Jesper, If you have 150 indexes, they you'll have 150 threads dedicated for these indexes. This is explicitly part of the design here. The idea is that we only trigger indexing for that thread when stuff that it actually care about are updated. Otherwise, it is a suspended thread and take very few resources. My current machine at this time is a laptop, and it is running about 2,800 threads. Chrome alone takes about 215 threads across about 8 processes.
The problem with sharing threads between indexes is that you get a LOT of stuff that you need to take care of, avoiding starvation,prioritization, etc. We can lean on the OS for handling all of that.
Note that indexes threads are not likely to be a problem. They are _static_, so once created, they pretty much don't come and go in this fashion. The problem is when you have stuff that does come and go. For example, handling replication connection is done via a dedicated thread, and that can drop and reconnect under load.
Right - but theoretically, under the right (or rather: wrong) circumstances, by creating one index and having one thread reserved for that index, I could "break the camel's back" as mentioned in the post by spawning a new thread? If you don't have to reinvent an OS scheduler and this is a rare event, it still sounds like a worthwhile tradeoff, I'm just trying to get a handle on how index threads are managed vs these pooled threads vs thread pool threads vs ...
Jesper, In such a case, when the index will try to create a new thread and the pool doesn't have any to give it, we explicitly check that there is enough breathing room for us to create a new thread. Otherwise we'll error and the index will be marked as errored, which is better than crashing the server.
That's great, much better than the behavior I assumed by reading the post on index threads! I would put that in bold somewhere to counter the intuition/assumption that creating an index at the wrong time could crash the server, which had me worried.
Jesper, Good idea, I updated the post
Comment preview