Ayende @ Rahien

It's a girl

Rule out the stupid stuff first

A customer reported that they have problems downloading large (10+ MB) attachments from RavenDB over the network. The problem wasn’t present when running locally.

The first thing I did was go and check how we are loading and sending the data to the user. But it seems fine, we were doing buffered reads from disk and streaming the results to the network.

I then wrote the simplest thing that I could think of to reproduce this issue:

class Program
{
    static void Main(string[] args)
    {
        var listener = new HttpListener
            {
                Prefixes = {"http://+:8080/"}
            };
        listener.Start();
        while (true)
        {
            var context = listener.GetContext();
            var sp = Stopwatch.StartNew();
            var chunks = int.Parse(context.Request.QueryString["chunks"]);
            var _4kb = new byte[4*1024];
            for (int i = 0; i < chunks; i++)
            {
                context.Response.OutputStream.Write(_4kb, 0, _4kb.Length);
            }
            context.Response.Close();
            var totalSize = (double) (_4kb.Length*chunks);
            Console.WriteLine("{0:#,#.##;;0} mb in {1:#,#}", Math.Round((totalSize / 1024) / 1024, 2), sp.ElapsedMilliseconds);
        }
    }
}

This is a simple HTTP server, which will stream to you as much data as you request, as fast as it can. The nice thing about it is that it is a pure in memory device, no IO / waiting required. So it should be able to saturate the network easily.

Using this, we can determine whatever the problem is really with RavenDB or something else.

The customer came back with a report that on their network, a 2MB request was taking over a minute. Considering that we are talking about internal networks, that sounded like we have identified the problem. There is something fishy going on with the network. Now it is in the hands of the sys admin, I just hope he is not this guy.

Tags:

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

Paul Hatcher
09/07/2012 11:36 AM by
Paul Hatcher

I had something similar on my home network - took ages to track down and turned out to be a hub that was on its way out.

Performance was fine with 1-2 ports occupied but degraded rapidly as more devices were added.

Jim Baltika
09/07/2012 12:21 PM by
Jim Baltika

I think the idea it self to store and server binary files in Raven DB for any enterprise level application is bad from beginning. We are using RavenDb with amazon S3 without any problems. Amazon S3 provides access security plus ability to access file with specific TTL (time to live). The best of all worlds :)

Gene Hughson
09/07/2012 01:09 PM by
Gene Hughson

These kinds of things are why I've always tried to remain a generalist at heart. Focusing intently on one single aspect makes it extremely likely that you'll run down blind alleys, not because they're probable but because they're what you understand.

Johannes Gustafsson
09/08/2012 10:04 PM by
Johannes Gustafsson

Was it really that simple? I admit, I actually felt quite naive to complain about it in the first place. The thing was that it was reproducible. I had 2 completely different environments, at different locations, set up by different people on different hardware, where I got the same behaviour. And it actually turned out (thanks to your sample) that there probably is some problem in HttpListener (or http.sys) when using 4k buffers.

Here is the original conversation: https://groups.google.com/d/topic/ravendb/EnI_2tjAwLg/discussion

Btw, the fix you made in the latest build made the problem go away, so thanks :-)

PS. A few people on the mailing list have complained about very slow download speeds for the silverlight UI. Since the code in Raven uses 4k buffers for this, my money is on that they have the same problem as I have. DS

Ayende Rahien
09/09/2012 04:56 AM by
Ayende Rahien

Johannes, It wasn't so simple. But wait for the next post in which I am talking about that.

Comments have been closed on this topic.