http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Magesh, KAE figured that out :-) http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment21http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment21Wed, 28 Apr 2010 10:08:03 GMTWould you care to post the solution? http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment20http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment20Wed, 28 Apr 2010 09:19:31 GMTI would use DirectoryInfo instead of Directory to get the file listing. It does less security checks when combined with FileInfo.OpenText it should speed the process when processing a significant number of files ... http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment19http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment19Tue, 27 Apr 2010 08:48:40 GMTI'd say it's the missing requestStream.Close within the loop. My guess would be that the Close operation would block the thread until the upload was complete so without it, (aside from the risk of running out of connections) your timer wouldn't reflect the actual upload performance. Hence, a false report of smashing performance. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment18http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment18Tue, 27 Apr 2010 07:30:35 GMTyou have to either consume or close the ResponseStream the very least that you have to do is call GetResponse().Close() the "fatal flaw" being that you are not closing the underlying HTTP connection at some point in time you are going to run out of available connections and be blocked waiting for an HTTP connection to timeout testing and even "production" running code will work given a small number of files http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment17http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment17Mon, 26 Apr 2010 16:56:32 GMT@Justin I think what you said above is correct, but when using Directory.GetFiles the order of the returned file names is not guaranteed, so using OrderBy is necessary. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment16http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment16Mon, 26 Apr 2010 08:02:25 GMTYou must call the Stream.Close method to close the stream and release the connection for reuse. Failure to close the stream causes your application to run out of connections. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment15http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment15Mon, 26 Apr 2010 07:51:58 GMTThe statement "Directory.GetFiles("Docs", searchPattern).OrderBy(x=>x)" By default, the GetFiles would usually return the files within the alphabetical order. Also, "GetFiles" is a blocking operation waiting for the entire array to be built up. Therefore, the Stopwatch would only start measuring after the more expensive operations began. This would also only measure the very last cycle in the foreach loop... not the overall response time of the ENTIRE method. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment14http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment14Mon, 26 Apr 2010 06:36:15 GMTThe http request is not submitted until the "GetResponse()" method of the HttpWebRequest class is invoked. The code in the blog post writes the data to the local stream and discards it without initiating the actual web request. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment13http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment13Mon, 26 Apr 2010 06:13:43 GMTYou never hit Stop on the stop watch. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment12http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment12Sun, 25 Apr 2010 17:34:22 GMTYou initialized your variable in the for loop so technically you are measuring only the last request and none of the file access or any of the other requests. var sp = Stopwatch.StartNew(); foreach (var file in Directory.GetFiles("Docs", searchPattern).OrderBy(x=>x)) { //var sp = Stopwatch.StartNew(); http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment11http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment11Sun, 25 Apr 2010 15:10:38 GMTHaven't tested it much (would definitely only work for sync, for instance), but: public class TimedWebClient : WebClient { public TimeSpan LastResponseTime { get; private set; } protected override WebResponse GetWebResponse(WebRequest request) { var stopWatch = Stopwatch.StartNew(); var response = base.GetWebResponse(request); this.LastResponseTime = stopWatch.Elapsed; return response; } } consume like: static void Main(string[] args) { var webClient = new TimedWebClient(); var url = " [http://www.google.com](http://www.google.com)"; webClient.DownloadString(url); Console.WriteLine("Downloading {0} took {1} ms", url, webClient.LastResponseTime.TotalMilliseconds); Console.ReadLine(); } http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment10http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment10Sun, 25 Apr 2010 13:18:27 GMTAs people pointed out, GetResponse actually fire the HTTP request. @Leonard - you should not reuse HttpWebRequest instances. Calling consecutive calls for GetResponse() will return a cached result of the first call, without re-issuing an HTTP request. btw, I'd use WebClient as long as you do not need to mingle too much with the returned Response in term of headers, certificates etc. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment9http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment9Sun, 25 Apr 2010 11:36:28 GMTShould this httpWebRequest be created in the loop? http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment8http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment8Sun, 25 Apr 2010 10:43:55 GMTMarco was right, you need to call GetResponse to perform the upload. If not, nothing is going to be uploaded. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment7http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment7Sun, 25 Apr 2010 10:23:26 GMTIf your trying to benchmark the overall performance of loading all the documents in the directory, then I'm guessing that the 'var sp = Stopwatch.StartNew();' should be outside the foreach loop. Otherwise the elapsed time is going to get reset on each iteration. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment6http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment6Sun, 25 Apr 2010 10:12:14 GMTAnother try (my last comment was flagged as spam): It might be that the server doesn't get the request until you invoke GetResponse due to buffering. Not related to the performance issue: Stopping the Stopwatch might be a good idea, or directly encapsulating the whole measurement. I did something similar here: [http://tinyurl.com/39op9j7](http://tinyurl.com/39op9j7)http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment5http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment5Sun, 25 Apr 2010 10:04:45 GMTUh oh... nope sorry, made a mistake.. The request is within a using block, so you're actually flushing it! It's Sunday morning here in Italy, and my brain started in Safe Mode today, LOL ;-) http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment4http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment4Sun, 25 Apr 2010 09:41:36 GMTForgot to specify HttpWebRequest.ContentLength before writing to the request stream? [msdn.microsoft.com/.../...quest.contentlength.aspx](http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.contentlength.aspx)http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment3http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment3Sun, 25 Apr 2010 09:30:17 GMTWhat @Marco said, plus you're not checking the response so you don't know if the request was successful. http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment2http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment2Sun, 25 Apr 2010 09:28:59 GMTUhm... perhaps you're not flushing the request stream and thus you're only measuring how long does it take to write data onto the stream's internal buffer? http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment1http://ayende.com/4471/how-to-get-really-fast-benchmarks#comment1Sun, 25 Apr 2010 09:17:50 GMT