The files in Directory.GetFiles("Docs","*.json") is the same directory as http://localhost:9090/bulk_docs, so you have an ever increasing filecount?

Wild guess is that it ran out of IP source port numbers?

Directory.GetFiles("Docs",".json") should be Directory.EnumerateFiles("Docs",".json") if you want to be Parallel.

Your testclient is sending more requests than the server can handle, maybe you're using some sort of queue on the server which overflows.

Wild guessing from my side.

Is it because the directory contains too many files?

Depending how your test is set up, could it be that Parallel ForEach and Raven DB are getting worker threads from the same thread pool?

hit OOM because the server was buffering all the post'ed files? It's gotta get the whole request (including file contents) into memory before passing it along AFAIK

What about the underlying database - maybe it had some concurrency problems - deadlocks, transaction timeouts, or run out of pooled connections?

A wild guess, doesthe Directory.GetFiles() method return a non-generic collection instead of a generic one? If so, you should cast it.

It effectively DoS'd the server by uploading too many files at the one time (there were more parallel threads going on the client than the server could accept, so they started to timeout).

90,000 files @ 100-200ms each, no limit on the degrees of parallelization - lemme guess you had around 8,000 threads active, with 1MB stack allocated each, and hit OOM?

Was it getting the same set of files ..

Richard:it is single-threaded

you hit max sockets,file descriptors per process/system.

msdn.microsoft.com/.../ms739169%28VS.85%29.aspx

the deafult is 64.

Unless you modify the registry to increase the limit WININET makes at most two distinct connections to the same remote host so you are only going to benefit from two threads. The other threads are going to block waiting for one of the two connections and if you have more than two processors in your system you are going to spin up more and more threads out of the .NET thread pool all of them blocking and taking up 1-2mb of virtual address space.

The thread-pool was spawning more and more threads (max by default is 250) because from its perspective the work was IO bound (waiting on the posts). It tries to saturate the CPU by spawning more threads.

Is PostTo doing an async post? I can't imagine how Parallel.ForEach would be bogging down the server since it limits the number of parallel tasks to the number of cores that you have. So if you are doing synchronous POST requests, it is only going to be posting 2-4 requests at a time, which is obviously not a lot.

Is it something to do with the fact that you're posting to the same uri over and over again?

I can imagine a scenario where at some point you decide to persist the documents, by recursively walking the documents to be written and because there are so many you end up blowing the stack somehow.

Does it have anything to do with TIME-WAIT? msdn.microsoft.com/en-us/library/ms819739.aspx

HttpWebRequest.KeepAlive was set to its default "true" value?

I am impressed because many creative solutions have been posted. By coincidence I faced the same issue 5min ago. It was the threadpool. Breaking in the debugger and executing ThreadPool.SetMaxThreads in the immediate window helped so I did not have to restart my long-running batch job.

The .NET thread pool does not create a thread unless there is a processor/core on your system that is doing nothing. If there are no processors available the thread pool puts your request in a queue. You shouldn't use with ThreadPool.SetMaxThreads. The problem is that a thread is created and it blocks immediately when WININET already has two connections to a given host. When it blocks the processor it was running on is freed and the thread pool takes a request out of its queue and schedules a thread. You end up with all these threads blocked each taking up 1-2mb of virtual address space and they are all waiting for the same WININET resource to become available.

This article discusses the WININET limit http://support.microsoft.com/kb/183110

Maybe it could be a problem with max http connections by server:

stackoverflow.com/.../improving-performance-of-...

Don't know if the Paralle.ForEach uses some sort of I/O port completion, but if so, I would think than blocking time waiting for socket reply ( the http request ) will be used, and do other file handles open and evantually run out of maximum file handles available. If I remember well, file handles are forced to some not so large count, in order to prevent buggy/malicious software to arm the system

Are you enumerating the entire contents of bulk_docs for every request to check your filename is unique?

Because you dumped 90,000 tasks into the Parallel Framework task scheduler?

Actually, it handled that really nicely.