http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Actually, it handled that really nicely. http://ayende.com/4475/challenge-what-killed-the-application#comment29http://ayende.com/4475/challenge-what-killed-the-application#comment29Fri, 30 Apr 2010 13:03:41 GMTBecause you dumped 90,000 tasks into the Parallel Framework task scheduler? http://ayende.com/4475/challenge-what-killed-the-application#comment28http://ayende.com/4475/challenge-what-killed-the-application#comment28Fri, 30 Apr 2010 12:20:05 GMTAre you enumerating the entire contents of bulk_docs for every request to check your filename is unique? http://ayende.com/4475/challenge-what-killed-the-application#comment27http://ayende.com/4475/challenge-what-killed-the-application#comment27Thu, 29 Apr 2010 07:47:05 GMTDon't know if the Paralle.ForEach uses some sort of I/O port completion, but if so, I would think than blocking time waiting for socket reply ( the http request ) will be used, and do other file handles open and evantually run out of maximum file handles available. If I remember well, file handles are forced to some not so large count, in order to prevent buggy/malicious software to arm the system http://ayende.com/4475/challenge-what-killed-the-application#comment26http://ayende.com/4475/challenge-what-killed-the-application#comment26Wed, 28 Apr 2010 20:59:54 GMTMaybe it could be a problem with max http connections by server: [stackoverflow.com/.../improving-performance-of-...](http://stackoverflow.com/questions/388908/improving-performance-of-multithreaded-httpwebrequests-in-net)http://ayende.com/4475/challenge-what-killed-the-application#comment25http://ayende.com/4475/challenge-what-killed-the-application#comment25Wed, 28 Apr 2010 18:11:13 GMTThis article discusses the WININET limit [http://support.microsoft.com/kb/183110](http://support.microsoft.com/kb/183110)http://ayende.com/4475/challenge-what-killed-the-application#comment24http://ayende.com/4475/challenge-what-killed-the-application#comment24Wed, 28 Apr 2010 17:11:14 GMTThe .NET thread pool does not create a thread unless there is a processor/core on your system that is doing nothing. If there are no processors available the thread pool puts your request in a queue. You shouldn't use with ThreadPool.SetMaxThreads. The problem is that a thread is created and it blocks immediately when WININET already has two connections to a given host. When it blocks the processor it was running on is freed and the thread pool takes a request out of its queue and schedules a thread. You end up with all these threads blocked each taking up 1-2mb of virtual address space and they are all waiting for the same WININET resource to become available. http://ayende.com/4475/challenge-what-killed-the-application#comment23http://ayende.com/4475/challenge-what-killed-the-application#comment23Wed, 28 Apr 2010 17:08:51 GMTI am impressed because many creative solutions have been posted. By coincidence I faced the same issue 5min ago. It was the threadpool. Breaking in the debugger and executing ThreadPool.SetMaxThreads in the immediate window helped so I did not have to restart my long-running batch job. http://ayende.com/4475/challenge-what-killed-the-application#comment22http://ayende.com/4475/challenge-what-killed-the-application#comment22Wed, 28 Apr 2010 14:21:02 GMTHttpWebRequest.KeepAlive was set to its default "true" value? http://ayende.com/4475/challenge-what-killed-the-application#comment21http://ayende.com/4475/challenge-what-killed-the-application#comment21Wed, 28 Apr 2010 13:35:56 GMTDoes it have anything to do with TIME-WAIT? [msdn.microsoft.com/en-us/library/ms819739.aspx](http://msdn.microsoft.com/en-us/library/ms819739.aspx)http://ayende.com/4475/challenge-what-killed-the-application#comment20http://ayende.com/4475/challenge-what-killed-the-application#comment20Wed, 28 Apr 2010 13:17:18 GMTIs it something to do with the fact that you're posting to the same uri over and over again? I can imagine a scenario where at some point you decide to persist the documents, by recursively walking the documents to be written and because there are so many you end up blowing the stack somehow. http://ayende.com/4475/challenge-what-killed-the-application#comment19http://ayende.com/4475/challenge-what-killed-the-application#comment19Wed, 28 Apr 2010 12:35:22 GMTIs PostTo doing an async post? I can't imagine how Parallel.ForEach would be bogging down the server since it limits the number of parallel tasks to the number of cores that you have. So if you are doing synchronous POST requests, it is only going to be posting 2-4 requests at a time, which is obviously not a lot. http://ayende.com/4475/challenge-what-killed-the-application#comment18http://ayende.com/4475/challenge-what-killed-the-application#comment18Wed, 28 Apr 2010 12:32:11 GMTThe thread-pool was spawning more and more threads (max by default is 250) because from its perspective the work was IO bound (waiting on the posts). It tries to saturate the CPU by spawning more threads. http://ayende.com/4475/challenge-what-killed-the-application#comment17http://ayende.com/4475/challenge-what-killed-the-application#comment17Wed, 28 Apr 2010 12:12:44 GMTUnless you modify the registry to increase the limit WININET makes at most two distinct connections to the same remote host so you are only going to benefit from two threads. The other threads are going to block waiting for one of the two connections and if you have more than two processors in your system you are going to spin up more and more threads out of the .NET thread pool all of them blocking and taking up 1-2mb of virtual address space. http://ayende.com/4475/challenge-what-killed-the-application#comment16http://ayende.com/4475/challenge-what-killed-the-application#comment16Wed, 28 Apr 2010 11:49:27 GMTRichard:it is single-threaded you hit max sockets,file descriptors per process/system. [msdn.microsoft.com/.../ms739169%28VS.85%29.aspx](http://msdn.microsoft.com/en-us/library/ms739169%28VS.85%29.aspx) the deafult is 64. http://ayende.com/4475/challenge-what-killed-the-application#comment15http://ayende.com/4475/challenge-what-killed-the-application#comment15Wed, 28 Apr 2010 11:49:15 GMTWas it getting the same set of files .. http://ayende.com/4475/challenge-what-killed-the-application#comment14http://ayende.com/4475/challenge-what-killed-the-application#comment14Wed, 28 Apr 2010 11:46:10 GMT90,000 files @ 100-200ms each, no limit on the degrees of parallelization - lemme guess you had around 8,000 threads active, with 1MB stack allocated each, and hit OOM? http://ayende.com/4475/challenge-what-killed-the-application#comment13http://ayende.com/4475/challenge-what-killed-the-application#comment13Wed, 28 Apr 2010 11:21:16 GMTIt effectively DoS'd the server by uploading too many files at the one time (there were more parallel threads going on the client than the server could accept, so they started to timeout). http://ayende.com/4475/challenge-what-killed-the-application#comment12http://ayende.com/4475/challenge-what-killed-the-application#comment12Wed, 28 Apr 2010 11:15:03 GMTA wild guess, doesthe Directory.GetFiles() method return a non-generic collection instead of a generic one? If so, you should cast it. http://ayende.com/4475/challenge-what-killed-the-application#comment11http://ayende.com/4475/challenge-what-killed-the-application#comment11Wed, 28 Apr 2010 11:10:57 GMTWhat about the underlying database - maybe it had some concurrency problems - deadlocks, transaction timeouts, or run out of pooled connections? http://ayende.com/4475/challenge-what-killed-the-application#comment10http://ayende.com/4475/challenge-what-killed-the-application#comment10Wed, 28 Apr 2010 11:09:02 GMThit OOM because the server was buffering all the post'ed files? It's gotta get the whole request (including file contents) into memory before passing it along AFAIK http://ayende.com/4475/challenge-what-killed-the-application#comment9http://ayende.com/4475/challenge-what-killed-the-application#comment9Wed, 28 Apr 2010 10:40:42 GMTDepending how your test is set up, could it be that Parallel ForEach and Raven DB are getting worker threads from the same thread pool? http://ayende.com/4475/challenge-what-killed-the-application#comment8http://ayende.com/4475/challenge-what-killed-the-application#comment8Wed, 28 Apr 2010 10:32:15 GMTIs it because the directory contains too many files? http://ayende.com/4475/challenge-what-killed-the-application#comment7http://ayende.com/4475/challenge-what-killed-the-application#comment7Wed, 28 Apr 2010 10:27:45 GMTYour testclient is sending more requests than the server can handle, maybe you're using some sort of queue on the server which overflows. Wild guessing from my side. http://ayende.com/4475/challenge-what-killed-the-application#comment6http://ayende.com/4475/challenge-what-killed-the-application#comment6Wed, 28 Apr 2010 10:25:54 GMTPeter, No, we haven't got that. But I run into this before. It usually only pop up using HTTPS, or authenticated connections, though. LS, Actually, no, we parallelize the action, not the enumeration, but thanks for letting me know about the new API http://ayende.com/4475/challenge-what-killed-the-application#comment5http://ayende.com/4475/challenge-what-killed-the-application#comment5Wed, 28 Apr 2010 10:10:05 GMTHenning, No, there is no association between the two. http://ayende.com/4475/challenge-what-killed-the-application#comment4http://ayende.com/4475/challenge-what-killed-the-application#comment4Wed, 28 Apr 2010 10:09:09 GMTDirectory.GetFiles("Docs","*.json") should be Directory.EnumerateFiles("Docs","*.json") if you want to be Parallel. http://ayende.com/4475/challenge-what-killed-the-application#comment3http://ayende.com/4475/challenge-what-killed-the-application#comment3Wed, 28 Apr 2010 09:23:31 GMTWild guess is that it ran out of IP source port numbers? http://ayende.com/4475/challenge-what-killed-the-application#comment2http://ayende.com/4475/challenge-what-killed-the-application#comment2Wed, 28 Apr 2010 09:20:49 GMTThe files in Directory.GetFiles("Docs","*.json") is the same directory as [http://localhost:9090/bulk_docs](http://localhost:9090/bulk_docs), so you have an ever increasing filecount? http://ayende.com/4475/challenge-what-killed-the-application#comment1http://ayende.com/4475/challenge-what-killed-the-application#comment1Wed, 28 Apr 2010 09:19:22 GMT