Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,971 | Comments: 44,508

filter by tags archive

Rule out the stupid stuff first


A customer reported that they have problems downloading large (10+ MB) attachments from RavenDB over the network. The problem wasn’t present when running locally.

The first thing I did was go and check how we are loading and sending the data to the user. But it seems fine, we were doing buffered reads from disk and streaming the results to the network.

I then wrote the simplest thing that I could think of to reproduce this issue:

class Program
{
    static void Main(string[] args)
    {
        var listener = new HttpListener
            {
                Prefixes = {"http://+:8080/"}
            };
        listener.Start();
        while (true)
        {
            var context = listener.GetContext();
            var sp = Stopwatch.StartNew();
            var chunks = int.Parse(context.Request.QueryString["chunks"]);
            var _4kb = new byte[4*1024];
            for (int i = 0; i < chunks; i++)
            {
                context.Response.OutputStream.Write(_4kb, 0, _4kb.Length);
            }
            context.Response.Close();
            var totalSize = (double) (_4kb.Length*chunks);
            Console.WriteLine("{0:#,#.##;;0} mb in {1:#,#}", Math.Round((totalSize / 1024) / 1024, 2), sp.ElapsedMilliseconds);
        }
    }
}

This is a simple HTTP server, which will stream to you as much data as you request, as fast as it can. The nice thing about it is that it is a pure in memory device, no IO / waiting required. So it should be able to saturate the network easily.

Using this, we can determine whatever the problem is really with RavenDB or something else.

The customer came back with a report that on their network, a 2MB request was taking over a minute. Considering that we are talking about internal networks, that sounded like we have identified the problem. There is something fishy going on with the network. Now it is in the hands of the sys admin, I just hope he is not this guy.


Comments

Paul Hatcher

I had something similar on my home network - took ages to track down and turned out to be a hub that was on its way out.

Performance was fine with 1-2 ports occupied but degraded rapidly as more devices were added.

Jim Baltika

I think the idea it self to store and server binary files in Raven DB for any enterprise level application is bad from beginning. We are using RavenDb with amazon S3 without any problems. Amazon S3 provides access security plus ability to access file with specific TTL (time to live). The best of all worlds :)

Gene Hughson

These kinds of things are why I've always tried to remain a generalist at heart. Focusing intently on one single aspect makes it extremely likely that you'll run down blind alleys, not because they're probable but because they're what you understand.

Johannes Gustafsson

Was it really that simple? I admit, I actually felt quite naive to complain about it in the first place. The thing was that it was reproducible. I had 2 completely different environments, at different locations, set up by different people on different hardware, where I got the same behaviour. And it actually turned out (thanks to your sample) that there probably is some problem in HttpListener (or http.sys) when using 4k buffers.

Here is the original conversation: https://groups.google.com/d/topic/ravendb/EnI_2tjAwLg/discussion

Btw, the fix you made in the latest build made the problem go away, so thanks :-)

PS. A few people on the mailing list have complained about very slow download speeds for the silverlight UI. Since the code in Raven uses 4k buffers for this, my money is on that they have the same problem as I have. DS

Ayende Rahien

Johannes, It wasn't so simple. But wait for the next post in which I am talking about that.

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Paying the rent online - about one day from now

There are posts all the way to Aug 03, 2015

RECENT SERIES

  1. Production postmortem (5):
    29 Jul 2015 - The evil licensing code
  2. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
  3. API Design (7):
    20 Jul 2015 - We’ll let the users sort it out
  4. What is new in RavenDB 3.5 (3):
    15 Jul 2015 - Exploring data in the dark
  5. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats