Ayende @ Rahien

It's a girl

RavenHQ & Amazon EC2 Outage

Update: The issue has been resolved, I’ll update on Sunday with full details about this.

Some RavenHQ customers may have noticed that they are currently unable to access their RavenHQ databases.

The underlying reason is an outage in Amazon US-EAST-1 region, which is where the ravenhq.com server and some of the data base servers are located.

Customers with replicated plans should see no disturbance of service, since this should trigger an automatic failover to the secondary node.

If you have any questions / require support, please contact us via our support forum: http://support.ravenhq.com/

You can see the status report from Amazon below. We managed to restore some service, but then lost it again (because of EBS timeouts, I suspect).

We are currently hard at work at bringing up new servers in additional availability zones, and we hope to restore full functionality as soon as possible.

image

Tags:

Published at

Originally posted at

Comments (6)

Geo Location & Spatial Searches with RavenDB–Part VI–Database Modeling

If you had sharp eyes, you might have noticed that in this code, I am actually using two different sessions:

We have the GeoSession, and we have the RavenSession.

The GeoSession is actually pointed at a different database, and it is a read only. In fact, here is how we use this:

image

As you can see, we create this on as needed basis, and we only dispose it, we never actually call SaveChanges().

So, those are the technical details, but what is the reasoning behind this?

Well, it is actually pretty simple. The GeoIP dataset is about 600 MB in size, and mostly it is about… well, geo location stuff. It is a very nice feature, but it is a self contained one, and not something that I really care for putting inside my app database. Instead, I have decided to go another way, and use a separate database.

That means that we have separation, at the data layer, between the different databases. It makes sense, about the only thing that we need from the GeoIP dataset is the ability to handle queries, and that is expressed only via GetLocationByIp, nothing more.

I don’t see a reason to make the app database bigger and more complex, or to have to support updates to the GeoIP dataset inside the app. This is a totally separate service. And having this in a separate database make it much easier to use this the next time that I want to use geo location. And it simplify my life right now with regards to maintaining and working with my current app.

In fact, we could have taken it even further, and not use RavenDB access to this at all. We can use REST calls to get the data out directly. We have chosen to still use the RavenDB Client, I’ll discuss exactly why we chose not to do that.

Reviewing Dinner Party – Nerd Dinner ported to RavenDB on RavenHQ

I got word about a  port of Nerd Dinner to RavenDB, Dinner Party (source, live demo), and I just had to check the code.

I reviewed Nerd Dinner itself in the past: Part I, Part II, so it is extra fun to see what happens when you move this to RavenDB. Note that at this point, I haven’t even looked at the code yet.

Here is the actual project (Just one project, I wholeheartedly approve):

image

Hm… where is the controllers folder?

Oh, wait, this isn’t an ASP.NET MVC application, it is a NancyFX application. I never actually look into that, so this should be interesting. Let us see if I can find something interesting, and I think we should look at the bootstrapper first.

There are several interesting things happening here. First, the application uses something called TinyIoC, which I am again, not familiar with. But it seems reasonable, and here is how it is initialized:

 protected override void ApplicationStartup(TinyIoC.TinyIoCContainer container, Nancy.Bootstrapper.IPipelines pipelines)
 {
     base.ApplicationStartup(container, pipelines);

     DataAnnotationsValidator.RegisterAdapter(typeof(MatchAttribute), (v, d) => new CustomDataAdapter((MatchAttribute)v));

     Func<TinyIoCContainer, NamedParameterOverloads, IDocumentSession> factory = (ioccontainer, namedparams) => { return new RavenSessionProvider().GetSession(); };
     container.Register<IDocumentSession>(factory);



     CleanUpDB(container.Resolve<IDocumentSession>());

     Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(IndexEventDate).Assembly, RavenSessionProvider.DocumentStore);
     Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(IndexUserLogin).Assembly, RavenSessionProvider.DocumentStore);
     Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(IndexMostPopularDinners).Assembly, RavenSessionProvider.DocumentStore);
     Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(IndexMyDinners).Assembly, RavenSessionProvider.DocumentStore);

     pipelines.OnError += (context, exception) =>
     {
         Elmah.ErrorSignal.FromCurrentContext().Raise(exception);
         return null;
     };
 }

All of which looks fine to me, except that I seriously don’t like the injection of the session. Why?

Because it means that if you have two components in the same request that needs a session, each will get his own session, instead of having a session per request. It also means that you can’t implement the “call SaveChanges() when the request is done without error” pattern, but that is more of a pet peeve than anything else.

Another thing to note is the multiple calls to IndexCreation.CreateIndexes. Remember, we have just one assembly here, and CreateIndexes operate on the assembly level, not on the individual index level. All of those can be removed but one (and it doesn’t matter which).

Lastly, we have the CleanupDB part. Dinner Party runs on Azure, and make use of RavenHQ. In order to stay within the limit of the RavenHQ free database, Dinner Party will do cleanups and delete old events if the db size goes over some threshold.

Okay, let us see where the real stuff is happening, and it seems to be happening in the Modules directory. I checked the HomeModule first, and I got:

public class HomeModule : BaseModule
{
    public HomeModule()
    {
        Get["/"] = parameters =>
        {
            base.Page.Title = "Home";

            return View["Index", base.Model];
        };

        Get["/about"] = parameters =>
        {
           
            base.Page.Title = "About";

            return View["About", base.Model];
        };
    }
}

I was worried at first about Page.Title (reminded me of ASPX pages), but it is just a default model that is defined in BaseModule. It is actually quite neat, if you think about it, check it out:

Before += ctx =>
{
    Page = new PageModel()
    {
        IsAuthenticated = ctx.CurrentUser != null,
        PreFixTitle = "Dinner Party - ",
        CurrentUser = ctx.CurrentUser != null ? ctx.CurrentUser.UserName : "",
        Errors = new List<ErrorModel>()
    };

    Model.Page = Page;

    return null;
};

I assume that Model is shared between Module and View, but I will check it shortly. I like how you can expose it to the view dynamically and have a strongly typed version in your code.

And yes, confirmed, the views are just Razor code, and they look like this:

image

Okay, enough with playing around, I’ll need to investigate NancyFX more deeply later on (especially since it can do self hosting), but right now, let us see how this is using RavenDB.

Let us start with the DinnerModule, a small snippet of it can be found here (this is from the ctor):

const string basePath = "/dinners";

Get[basePath + Route.AnyIntOptional("page")] = parameters =>
{

    base.Page.Title = "Upcoming Nerd Dinners";
    IQueryable<Dinner> dinners = null;

    //Searching?
    if (this.Request.Query.q.HasValue)
    {
        string query = this.Request.Query.q;

        dinners = DocumentSession.Query<Dinner>().Where(d => d.Title.Contains(query)
                || d.Description.Contains(query)
                || d.HostedBy.Contains(query)).OrderBy(d => d.EventDate);
    }
    else
    {
        dinners = DocumentSession.Query<Dinner, IndexEventDate>().Where(d => d.EventDate > DateTime.Now.Date)
            .OrderBy(x => x.EventDate);
    }

    int pageIndex = parameters.page.HasValue && !String.IsNullOrWhiteSpace(parameters.page) ? parameters.page : 1;



    base.Model.Dinners = dinners.ToPagedList(pageIndex, PageSize);

    return View["Dinners/Index", base.Model];

};

I am not sure that I really like this when you effectively have methods within methods, and many non trivial ones.

The code itself seems to be pretty nice, and I like the fact that it makes use of dynamic in many cases to make things easier (for Query or to get the page parameter).

But where does DocumentSession comes from? Well, it comes from PersistentModule, the base class for DinnerModule, let us take a look at that:

public class PersistModule : BaseModule
{
    public IDocumentSession DocumentSession
    {
        get { return Context.Items["RavenSession"] as IDocumentSession; }
    }

    public PersistModule()
    {
    }

    public PersistModule(string modulepath)
        : base(modulepath)
    {
    }
}

And now I am confused, so we do have session per request here? It appears that we do, there is a RavenAwareModuleBuilder, which has the following code:

if (module is DinnerParty.Modules.PersistModule)
{
    context.Items.Add("RavenSession", _ravenSessionProvider.GetSession());
    //module.After.AddItemToStartOfPipeline(ctx =>
    //{
    //    var session =
    //        ctx.Items["RavenSession"] as IDocumentSession;
    //    session.SaveChanges();
    //    session.Dispose();
    //});
}

I withdraw my earlier objection. Although note that the code had at one point automatic session SaveChanges(), and now it no longer does.

Another common pet issue in the code base, there is a lot of code that is commented.

Okay, so now I have a pretty good idea how this works, let us see how they handle writes, in the Dinner case, we have another class, called DinnerModuleAuth, which is used to handle all writes.

Here is how it looks like (I chose the simplest, mind):

Post["/delete/" + Route.AnyIntAtLeastOnce("id")] = parameters =>
    {
        Dinner dinner = DocumentSession.Load<Dinner>((int)parameters.id);

        if (dinner == null)
        {
            base.Page.Title = "Nerd Dinner Not Found";
            return View["NotFound", base.Model];
        }

        if (!dinner.IsHostedBy(this.Context.CurrentUser.UserName))
        {
            base.Page.Title = "You Don't Own This Dinner";
            return View["InvalidOwner", base.Model];
        }

        DocumentSession.Delete(dinner);
        DocumentSession.SaveChanges();

        base.Page.Title = "Deleted";
        return View["Deleted", base.Model];
    };

My only critique is that I don’t understand why we would need to explicitly call SaveChanges instead.

Finally, a bit of a critique on the RavenDB usage, the application currently uses several static indexes: IndexEventDate, IndexMostPopularDinners, IndexMyDinners and IndexUserLogin.

The first three can be merged without any ill effects, I would create this, instead:

public class Dinners_Index : AbstractIndexCreationTask<Dinner>
{
    public Dinners_Index()
    {
        this.Map = dinners =>
                   from dinner in dinners
                   select new
                   {
                       RSVPs_AttendeeName = dinner.RSVPs.Select(x => x.AttendeeName),
                       RSVPs_AttendeeNameId = dinner.RSVPs.Select(x => x.AttendeeNameId),
                       HostedById = dinner.HostedById,
                       HostedBy = dinner.HostedBy,
                       DinnerID = int.Parse(dinner.Id.Substring(dinner.Id.LastIndexOf("/") + 1)),
                       Title = dinner.Title,
                       Latitude = dinner.Latitude,
                       Longitude = dinner.Longitude,
                       Description = dinner.Description,
                       EventDate = dinner.EventDate,
                       RSVPCount = dinner.RSVPs.Count,
                   };
    }
}

This serve the same exact function, but it only has one index. In general, we prefer to have bigger and fewer indexes than smaller and more numerous indexes.

Tags:

Published at

Originally posted at

Comments (13)

Unprofessional code: Answer

In my previous post, I mentioned that I don’t like this code, that it lacked an important *ities, and I asked what was the problem with it.

image_thumb

The answer is that it isn’t easily debuggable. Consider the case where we have a support call from a user “I am in France but I see courses that are running in Australia”.

How do you debug something like that, and how can you work with the user on that.

In this case, adding something like this is enough to make the system much nicer to work with under those scenarios:

image

This is an internal detail, never expose externally (until this blog post, obviously), but it allows us to override what the system is doing. We can actually pretend to be another user and see what is going on. Without the need of actual debugging any code. That means that you can try things out, in production. It means that you don’t have to work hard to replicate a production environment, get all of the  production data and so on.

Thinking like that is important, it drastically reduces your support burden over time.

Tags:

Published at

Originally posted at

Comments (33)

Unprofessional code

This code bugs me.

image

It isn’t wrong, and it is going to produce the right result. But is ain’t pro code, in the sense that this code lacks an important *ities.

Just to cut down on the guessing, I marked the exact part that bugs me. Can you see the issue?

Tags:

Published at

Originally posted at

Comments (41)

Geo Location & Spatial Searches with RavenDB–Part V-Spatial Searching

So, all of this have gone quite far. We have seen that we can quite easily go from having the user’s IP address to figuring out its location using our Geo Location database.

The next step is to actually do something about it. Usually, when doing geo location, you care to know what the human readable name for the user’s location is, but in our case, what we most care about is the user’s physical location, so it is lucky that the geo location database that we found also include longitude and latitude information.

With that, we can define the following index, on our events. The Longitude & Latitude information is actually calculated by the browser using the Google Geocoder API, so we just plug in the address, and the site does the rest by figuring out where on the globe we are.

This index allows us to search by spatial query as well as by time:

image

Using that, we can do:

image

First, we try to find the location of the user based on the user IP, then we are making a spatial query on top of the events, based on the user location.

What we are doing essentially is asking, “show me the next 2 future events within 200 miles of the user’s location”. Just a tiny little bit of code, but it produces this:

image

And hopefully, this will narrow things down for you to the obvious: “Of course I am going to the next RavenDB Course”!

Tags:

Published at

Originally posted at

Comments (7)

Get off my bed! And a RavenDB Course Discounts for Munich and Chicago

I can’t sleep, due to a literal doggie pile on my bed. Hence, I lay awake and ponder things like pricing strategies and other interesting topics.

image

In fact, as you can see, I put both dogs to sleep when taking about this.

The big one is a female German Shepherd, named Arava, and the smaller one is a charming fellow named Oscar.

And this is relevant to the discussion on a technical blog exactly how?

Well, as I said, Arava is a German Shepherd, and I have an upcoming RavenDB Course in Munich next month. If you go and register to the course, you can use Arava as the coupon code to get a 25% discount.

And before I forget Oscar (and hence get the biggest puppy eyes stare in the world), if you go and register to the Chicago RavenDB Bootcamp in late August, and use Oscar as the coupon code, you’ll get a 20% discount (what can I do, he is smaller).

The offer will remain open until I wake up, you have some time, I still need to find some way to get them off the bed : –).

Tags:

Published at

Originally posted at

Comments (5)

Geo Location & Spatial Searches with RavenDB–Part IV-Searching

Now we have all of the data loaded in, we need to be able to search on it. In order to do that, we define the following index:

image

It is a very simple one, mapping the start and end of each range for each location.

The next step is actually doing the search, and this is where we run into some issues. The problem was with the data:

image

Let us take the first range and translate that to IP addresses in the format that you are probably more used to:

Start: 0.177.195.68 End: 255.177.195.68

Yep, it is little endian vs. big endian here to bite us once more.

It took me a while to figure it out, I’ll admit. In other words, we have to reverse the IP address before we can search on it properly. Thankfully, that is easily done, and we have the following masterpiece:

image

The data source that we have only support IPv4, so that is what we allow. We reverse the IP, then do a range search based on this.

Now we can use it like this:

var location = session.GetLocationByIp(IPAddress.Parse("209.85.217.172"));

Which tells us that this is a Mountain View, CA, USA address.

More importantly for our purposes, it tells us that this is located at: 37.4192, -122.0574 We will use that shortly to do spatial searches for RavenDB events near you, which I’ll discuss in my next post.

Oh, and just for fun. You might remember that in previous posts I mentioned that MaxMind (the source for this geo location information) had to create its own binary format because relational databases took several second to process each query?

The query above completed in 53 milliseconds on my machine, without any tuning on our part. And that is before we introduce caching into the mix.

Tags:

Published at

Originally posted at

Comments (10)

Geo Location & Spatial Searches with RavenDB–Part III-Importing

The following are sample from the data sources that MaxMind provides for us:

image

image

The question is, how do we load them into RavenDB?

Just to give you some numbers, there are 1.87 million blocks and over 350,000 locations.

Those are big numbers, but still small enough that we can work with the entire thing in memory. I wrote a quick & ugly parsing routines for them:

public static IEnumerable<Tuple<int, IpRange>> ReadBlocks(string dir)
{
    using (var file = File.OpenRead(Path.Combine(dir, "GeoLiteCity-Blocks.csv")))
    using (var reader = new StreamReader(file))
    {
        reader.ReadLine(); // copy right
        reader.ReadLine(); // header

        string line;
        while ((line = reader.ReadLine()) != null)
        {
            var entries = line.Split(',').Select(x => x.Trim('"')).ToArray();
            yield return Tuple.Create(
                int.Parse(entries[2]),
                new IpRange
                {
                    Start = long.Parse(entries[0]),
                    End = long.Parse(entries[1]),
                });
        }
    }
}

public static IEnumerable<Tuple<int, Location>> ReadLocations(string dir)
{
    using (var file = File.OpenRead(Path.Combine(dir, "GeoLiteCity-Location.csv")))
    using (var reader = new StreamReader(file))
    {
        reader.ReadLine(); // copy right
        reader.ReadLine(); // header

        string line;
        while ((line = reader.ReadLine()) != null)
        {
            var entries = line.Split(',').Select(x => x.Trim('"')).ToArray();
            yield return Tuple.Create(
                int.Parse(entries[0]),
                new Location
                {
                    Country = NullIfEmpty(entries[1]),
                    Region = NullIfEmpty(entries[2]),
                    City = NullIfEmpty(entries[3]),
                    PostalCode = NullIfEmpty(entries[4]),
                    Latitude = double.Parse(entries[5]),
                    Longitude = double.Parse(entries[6]),
                    MetroCode = NullIfEmpty(entries[7]),
                    AreaCode = NullIfEmpty(entries[8])
                });
        }
    }
}

private static string NullIfEmpty(string s)
{
    return string.IsNullOrWhiteSpace(s) ? null : s;
}

And then it was a matter of bringing it all together:

var blocks = from blockTuple in ReadBlocks(dir)
             group blockTuple by blockTuple.Item1
             into g
             select new
             {
                 LocId = g.Key,
                 Ranges = g.Select(x => x.Item2).ToArray()
             };

var results =
    from locTuple in ReadLocations(dir)
    join block in blocks on locTuple.Item1 equals block.LocId into joined
    from joinedBlock in joined.DefaultIfEmpty()
    let _ = locTuple.Item2.Ranges = (joinedBlock == null ? new IpRange[0] : joinedBlock.Ranges)
    select locTuple.Item2;

 

The advantage of doing things this way is that we only have to write to RavenDB once, because we merged the results in memory. That is why I said that those are big results, but still small enough for us to be able to process them easily in memory.

Finally, we wrote them to RavenDB in batches of 1024 items.

The entire process took about 3 minutes and wrote 353,224 documents to RavenDB, which include all of the 1.87 million ip blocks in a format that is easy to search through.

In our next post, we will discuss actually doing searches on this information.

Tags:

Published at

Originally posted at

Comments (2)

Geo Location & Spatial Searches with RavenDB–Part II–Modeling

I mentioned in the previous post that the data is coming in a highly relational format, and that the MaxMind advice is to not load this into a relational database, because then queries take a long time. Instead, they have their own efficient binary format.

I decided to put this into RavenDB, and that brought the question, how are we actually going to work with this? How do we model the data?

In our code, we defined the following, which I guess would pretty closely match how you model the data in a relational database:

image

Except…

In this case, we are actually storing the whole thing as a single entry, like so:

image

This allows us to query things efficiently and easily. But before we can get there, we need to actually load the data in, which is the topic of my next post.

Tags:

Published at

Originally posted at

Comments (4)

Geo Location & Spatial Searches with RavenDB–Part I–Setup

One of the things that we are putting into the new version of the RavenDB Website is some smarts about courses. We want to be able to figure out where a visitor is coming from, and show him relevant courses near him.

This is a fairly standard feature, I guess. But I think we solved this in an interesting fashion. First, we had to figure out how to know where a visitor is coming from. This is done via geo location. There are plenty of geo location API, such as:

image

The problem is that a service like that is limited to ~1000 queries per hour, and we didn’t want that. Other option include paid services, again, with either hard limits on the number of queries per hour or an open credit line. Neither option was very interesting to us.

Instead, we thought we would try something out. Max Mind is offering a database that can be used for geo location purposes. They have two options, a free database and a paid option. I took the free version for a spin.

Here is the GoeLiteCity-Blocks information:

image

Note that we have repeated rows per location, which allows us to map multiple ranges for each location.

And finally, we have GoLiteCity-Location, which map a location id to an actual location:

image

Anyone who used a relational database should be familiar with the data format, right?

I found it really interesting that MaxMind state on their site:

Note that queries made against the CSV data imported into a SQL database can take up to a few seconds. If performance is an issue, the binary format is much faster, and can handle thousands of lookups per second.

They actually have a custom binary format to allow faster queries, but I think that I can do something about this. In the next post in this series, I’ll talk about how to import this data into RavenDB and the model concerns that we have with it.

Tags:

Published at

Originally posted at

Comments (11)

On Umbraco’s NHibernate’s Pullout

I was asked to comment on this topic:

image

Here is the link to the full article, but here is the gist:

Performance.
The culprit of v5 is our usage of NHibernate. This doesn't mean that NHibernate is evil, just that it has turned out to be the wrong fit for our project (in part due to our architecture and db schema, in part due to lack of proper NHibernate experience on the team). It's a badly tamed NHibernate that is caused the many N+1 SQL statements. The long term solution is to replace NHibernate with native, optimized SQL calls but it's a bigger operation that we'll look into over the summer. The short team solution is to continue with finding as many bottlenecks as we can as tune those, combined with introducing a Lucene based cache provider which will cache queries and results which will survive app pool restarts (much like the v4 XML cache). The architecture has been prepared for the latter and we'll have this in place by the end of May.


Another thing that causes massive performance issues is that we're seeing a trend where people are calling ToList() on razor queries to compensate for the lack of LINQ features in our implementation. Unfortunately, this causes most (if not all) of the database to be loaded into memory, naturally causing big performance issues (and an enormous number of N+1 SQL calls). This leads us to...

And a while later:

V5 couldn't be fixed. We needed to completely rewrite the entire business layer to ever get v5 into being the type of product people expect from Umbraco

I marked the parts that I intend to respond to in yellow, green highlight is for some additional thoughts later on, keep it in mind.

Now, to be fair, I looked at Umbraco a few times, but I never actually used it, and I never actually dug deep.

The entire problem can be summed in this page, from the Umbraco documentation site. This is the dynamic model that Umbraco exposes to the views to actually generate the HTML. From the docs:

DynamicModel is a dynamic object, this gives us a number of obvious benefits, mostly a much lighter syntax for accessing data, as well as dynamic queries, based on the names of your DocumentTypes and their properties, but at the same time, it does not provide intellisense.

Any time you provide the view with a way to generate queries, it is a BIG issue. For the purpose of discussion, I’ll focus on the following example,  also taken from Umbraco docs:

   1:  @inherits PartialViewMacroPage
   2:  @using Umbraco.Cms.Web
   3:  @using Umbraco.Cms.Web.Macros
   4:  @using Umbraco.Framework
   5:   
   6:  @{
   7:      @* Get the macro parameter and check it has a value - otherwise set to empty hive Id *@
   8:      var startNodeID = String.IsNullOrEmpty(Model.MacroParameters.startNodeID) ? HiveId.Empty.ToString() : Model.MacroParameters.startNodeID;
   9:  }
  10:   
  11:  @* Check that startNodeID is not an empty HiveID AND also check the string is a valid HiveID *@
  12:  @if (startNodeID != HiveId.Empty.ToString() && HiveId.TryParse(startNodeID).Success)
  13:  {
  14:      @* Get the start node as a dynamic node *@
  15:      var startNode = Umbraco.GetDynamicContentById(startNodeID);
  16:   
  17:      @* Check if the startNode has any child pages, where the property umbracoNaviHide is not True *@    
  18:      if (startNode.Children.Where("umbracoNaviHide != @0", "True").Any())
  19:      {
  20:          <ul>
  21:              @* For each child page under startNode, where the property umbracoNaviHide is not True *@
  22:              @foreach (var page in startNode.Children.Where("umbracoNaviHide != @0", "True"))
  23:              { 
  24:                  <li><a href="@page.Url">@page.Name</a></li>
  25:              }
  26:          </ul>
  27:      }    
  28:  }

This is a great example of why you should never do queries from the views. Put simply, you don’t have any idea what is going to happen. Let us start going through the onion, and see where we get.

Umbraco.GetDynamicContentById – Is actually UmbracoHelper.GetDynamicContentById, which looks like this:

image

I mean, I guess I could just stop right here, look at the first line, and it should tell you everything. But let us dig deeper. Here is what GetById looks like:

image

Note that with NHibernate, this is a worst practice, because it forces a query, rather than allow us to use the session cache and a more optimized code paths that Load or Get would use.

Also, note that this is generic on top of an IQueryable, so we have to go one step back and look at Cms().Content, which is implemented in:

image

I just love how we dispose of the Unit of Work that we just created, before it is used by any client. But let us dig a bit deeper, okay?

image

image

image

And we still haven’t found NHibernate! That is the point where I give up, I am pretty sure that Umbraco 5 is using NHibernate, I just can’t find how or where.

Note that in those conditions, it is pretty much settled that any optimizations that you can offer isn’t going to work, since you don’t have any good way to apply it, it is so hidden behind all those layers.

But I am pretty sure that it all end up with a Linq query on top of  a session.Query<Content>(), so I’ll assume that for now.

So we have this query in line 15, to load the object, then, on line 18, we issue what looks like a query:

if (startNode.Children.Where("umbracoNaviHide != @0", "True").Any())

We actually issue two queries here, one to load the Children collection, and the second in the Where(). It took a while to figure out what the Where was, but I am pretty sure it is:

image

Which translate the string to a Linq query, which then have to be translated to a string again. To be perfectly honest, I have no idea if this is executed against the database ( I assume so ) or against the in memory collection.

Assuming it is against the database, then we issue the same query again, two lines below that.

And this is in a small sample, think about the implications for a more complex page. When you have most of you data access so abstracted, it is very hard to see what is going on, let alone do any optimizations. And when you are doing data access in the view, the very last thing that the developer has in his mind is optimizing data access.

This is just setting up for failure. Remember what I said about: replace NHibernate with native, optimized SQL calls, this is where it gets interesting. Just about every best practice of NHibernate was violated here, but the major problem is not the NHibernate usage. The major problem is that there is very little correlation between what is happening and the database. And the number of queries that are likely to be required to generate each page are truly enormous.

You don’t issues queries from the view, that is the #1 reason for SELECT N+1, and when you create a system where that seems to be the primary mode of operation, that is always going to cause you issues.

Replace NHibernate with optimized SQL calls, but you are still making tons of direct calls to the DB from the views.

I actually agree with the Umbraco decision, it should be re-written. But the starting point isn’t just the architectural complexity. The starting point should be the actual interface that you intend to give to the end developer.

Tags:

Published at

Originally posted at

Comments (40)

Tertiary includes in RavenDB

For a while, I was absolutely against this, in fact, I refused to implement this several times when asked, because it is complicate to do and indicate design problems in your domain model.

During a course today, I found out that I might want to refuse, but the feature is actually in the product for the last two+ years, we just didn’t know about this.

But what are tertiary includes in the first place?

Let us imagine that we have something like this:

image

Now, I want to load an order, but also show its Customer and Products. Those are secondary includes, and are very easy to do with RavenDB:

var orders = session.Query<Order>()
.Include("Customer") .Include("Lines,Product") .ToList();

There is also a strongly typed option for this, of course.

What this does is instruct RavenDB to load into the session the Customer and the associated Products into the session, so when you do something like this:

var cust = session.Load<Customer>(order.Customer);

The value will be loaded from the session cache, without going to the sever. As I said, this is a feature that we have had for quite a while, and it is a really nice one, because it drastically reduce the number of queries that you have to make.

The problem is that some people want to take it one step further, they want to be able to search on an Order, but also load the Location of a Product. I don’t really like this, and as I said, when asked for this feature, I consistently said that it isn’t there because it represent a remnant of relational thinking in your design.

But as it turned out, we do support this, although quite by accident.

The reason is quite simple, we evaluate Includes only after we evaluate the TranformResults function. Which means that the TranformResults function gets to choose whatever you want to include. Here is how it works:

TranformResults  = (database, results) =>
   from result in results
   select new 
   {
     Order = result,
     Locations = result.SelectMany(x=>x.Lines).Select(x=>database.Load<Product>(x.Product).Location)
   }

And then, you can just ask to Include(“Locations”), and you are pretty much set.

Except, that this is a really awkward thing to do, and I don’t really like it at all.

Sure, I don’t like this feature, but people will use it, and if it already there, we might as well make it elegant. Therefor, we now have the option of doing:

TranformResults  = (database, results) =>
   from result in results
   let _ = database.Include(result.SelectMany(x=>x.Lines).Select(x=>x.Product))
   select result;

I think you’ll agree that this is much nicer all around, this tells the server to include the data, without us needing to explicitly ask this from the client.

Tags:

Published at

Originally posted at

Comments (11)

Entities Associations: Point in Time vs. Current Associations

Having just finished giving three courses (2 on RavenDB and 1 on NHibernate), you might want to say that I have a lot of data stuff on my mind. Teaching is always a pleasure to me, and one of the major reasons for that is that I get to actually learn a lot whenever I teach.

In this case, in all three courses, we run into an issue with modeling associations. For the sake of the example, let us talk about employees and paychecks. You can see the model below:

image

Do you note the blue lines? Those represent Employee reference, but while they are both referencing the same employee, they are actually quite different associations.

The Manager association is a Current Association. It is just a pointer to the managing employee. What does this means?

Let us say that the manager of a certain employee changed her name. In that scenario, when we look at the current employee record, we should see the updated employee manager name. In this case, we are always interested in the current status.

On the other hand when looking at the paycheck PaidTo reference to an employee, we have something all together different. We have a reference no to the current employee record, but to the employee record as it was at a certain point in time. If the employee in question change his name, that paycheck was issued to Mr. Version One, not to Mr. Version Two, even though the name has been changed.

when dealing with associations, it is important to distinguish between the two options, as each require different way of working with the association.

Code crimes, because even the Law needs to be broken

In 99.9% of the cases, if you see this, it is a mistake:

image

But I recently came into a place where this is actually a viable solution to a problem. The issue occurs deep inside RavenDB. And it boils down to a managed application not really having any way of actually assessing how much memory it is using.

What I would really like to be able to do is to specify a heap (using HeapCreate) and then say: “The following objects should be created on this heap”, which would allow to me to control exactly how much memory I am using for a particular task. In practice, that isn’t really possible (if you know how, please let me know).

What we do instead is to use as much memory as we can, based on the workload that we have. At certain point, we may hit an Out of Memory condition, which we take to mean, not that there is no more memory, but the system’s way of telling us to calm down.

By the time we reached the exception handler, we already lost all of the current workload, that means that now there is a LOT of garbage around for the GC. Before, when it tried to cleanup, we were actually holding to a lot of that memory, but not anymore.

Once an OOME happened, we can adjust our own behavior, to know that we are consuming too much memory and should be more conservative about our operations. We basically reset the clock back and become twice as conservative about using more memory. And if we get another OOME? We become twice as conservative again Smile, and so on.

Eventually, even under high workloads and small amount of memory, we complete the task, and we can be sure that we make effective use of the system’s resources available to us.

RavenDB Training

Demand for RavenDB training has been going up, and we are actually running around (pretty much world wide) trying to catch up.

After completing a course in Stockholm a couple of weeks ago, we have a new course in Stockholm in October, which you can register to using the following link.

I’ll also be doing a course in Munich, Germany, early next month. You can register to that here.

And, of course, we have the already scheduled courses, I mean, just take a look:

And on top of that, we have been doing some really cool stuff that I can’t hardly wait to talk about this, as part of our upcoming 1.2 release for RavenDB.

Tags:

Published at

Originally posted at

Comments (8)

Why I LOVE ReSharper

image

I mean, just look at this. This is magic.

More to the point, think about what this means to try to implement something like that. I wouldn’t know where to even start.

Tags:

Published at

Originally posted at

Comments (17)

Assuming that the laws of physics no longer apply, we can build this

This is a reply to a post by Frans Bouma, in which he asks for:

…loud vendors should offer simply one VM to me. On that VM I run the websites, store my DB and my files. As it's a virtual machine, how this machine is actually ran on physical hardware (e.g. partitioned), I don't care, as that's the problem for the cloud vendor to solve. If I need more resources, e.g. I have more traffic to my server, way more visitors per day, the VM stretches, like I bought a bigger box. This frees me from the problem which comes with multiple VMs: I don't have any refactoring to do at all: I can simply build my website as if it runs on my local hardware server, upload it to the VM offered by the cloud vendor, install it on the VM and I'm done.

Um… no.

Go ahead and read the whole post, it is interesting. But the underlying premise that is rely on is flawed. It is like starting out with assuming that since TCP/IP contains no built in prohibition to send data faster than light, the cloud providers can and should create networks that can send data faster than light. After all, I can show a clear business case for the reduced ping time, and that is certainly something that can be abstracted from my application.

What aren’t those bozos doing that?

Well, the answer to that is that it just ain’t possible. There are several minor problems along the way. The CAP theorem, to start with, but even if we ignore that aspect of the problem, there are also the fallacies of distributed computing.

According to Frans’ premise, we can have a single VM that can scale up to as many machines as is needed, without any change required to the system. Let us start with Frans’ answer to the actual scope of the problem:

But what about memory replication and other problems?

This environment isn't simple, at least not for the cloud vendor. But it is simple for the customer who wants to run his sites in that cloud: no work needed. No refactoring needed of existing code. Upload it, run it.

Um.. no.

Let us take a look at a few pieces of code, and see what is going to happen to then in Frans’ cloud environment. For example, let us take a look at this:

var tax = 0;
foreach(var item in order.Items)
{
  tax += item.CalculateTax();   
}
order.Tax = tax;

Problem, because of the elasticity of the VM, we actually spread things around so each of the items in the order collection is located in another physical machine. This is, of course, completely transparent to the code. But that means that each loop iteration is actually doing a network call behind the scene.

OR/M users are familiar with this as the SELECT N+1 problem, but in this case, you have a potential problem on every memory access. Network attached memory isn’t new, you can read about it in OS books and it is a nice theoretical idea, but it is just isn’t going to work, because you actually care about the speed of accessing the data.

In fact, we have many algorithms that were changed specifically to be able to take advantage of cache lines, L1 & L2 cache, etc. Because that has a major increase in the system performance, and that is only on a single machine. Trying to imagine a transparent network memory is futile, you actually care about memory access speed, a lot.

But let us talk about another aspect, I want to make have an always incrementing order id number. So I do:

Interlocked.Increment(ref lastOrderId);

All well and good when running on a single machine, but how should the VM make it work when running on multiple machines?

And remember, this call actually translate to a purpose built assembly instruction (XADD or one of its friends). In this case, you need to do this across the network, and touch as many machines as your system currently runs on.

But the whole point here is to allow us to rapidly generate a new number. This has now turned into a total mess in terms of performance.

What about parallel computing, for that matter?

var results = new Result[items.Length];
Parallel.For(items, (item, i) => 
{
    results[i] = item.Calculate();
});

I have many items, and I want to be able to compute the result in parallel, so I run this fairly standard code. But we are actually going to execute this on multiple threads, so this get scheduled on several different machines. But now you have to copy the results buffer to all of those machines, as well as any related state that they have, then copy it back out when it is done, then somehow merge the different changes made by different systems into a coherent whole.

Good luck with that.

I could go on, but I think that you get the point by now.

And we haven’t talked about the error condition yet. What happen if my system is running on 3 machines, and one of them goes down (power outage, hardware failure, etc)? 3rd of my memory, ongoing work and a lot of stuff just got lost. For that matter, I might have (actually, probably have) dangling references to memory that used to be on the failed machines, so the other two systems are likely to hit this inaccessible memory and fail themselves.

So.. no, this idea is a pipe dream, it isn’t going to work, not because of some evil plot by dastardly folks conspiring to make your life harder, but for the simple reason that it is easier to fly by flapping your arms.

Tags:

Published at

Originally posted at

Comments (19)

The HIGH cost of ConcurrentBag in .NET 4.0

I got some strange results when using concurrent collections, so I decided to try to track it down, and wrote the following code:

var count = ?;
var list = new List<object>(count);
var sp = Stopwatch.StartNew();
for (int i = 0; i < count; i++)
{
    list.Add(new ConcurrentBag<int>());
}
sp.Stop();
Console.WriteLine("{0} {2} items in {1:#,#;;0}ms = {3:#,#;;0}ms per item",
    sp.Elapsed, sp.ElapsedMilliseconds, count, sp.ElapsedMilliseconds / count);

And then I started to play with the numbers, and it is not good.

  • 10 items in 2ms = 0ms per item

This is incredibly high number, you have to understand. Just to compare, List<int> takes 8 ms to create 100,000 items.

Let us see how it works when we use more of this.

  • 100 items in 5ms = 0ms per item
  • 1,000 items in 37ms = 0ms per item
  • 10,000 items in 2,319ms = 0ms per item

Note the numbers, will you?

1,000 items in 37 ms, but 10,000 items? 2.3 seconds!

  • 20,000 items in 21,331ms = 1ms per item

And doubling the amount took ten times as long?

  • 25,000 items in 32,588ms = 1ms per item

And at this point, I stopped trying, because I didn’t have the patience.

Note that the other concurrent collection, ConcurrentStack, ConcurrentQueue and ConcurrentDictionary do not suffer from the same problem.

I contacted Microsoft about this, and this is already resolved in .NET 4.5. The underlying issue was that ThreadLocal, which ConcurrentBag uses, didn’t expect to have a lot of instances. That has been fixed, and now can run fairly fast.

Tags:

Published at

Originally posted at

Comments (10)

Truth in advertising

So I am at a client site discussing things about their new version of the software. And while it is easy to promise things, we did a POC to make sure that things will work out.

Because I know users, I made sure to have this ready:

image

I do software behavior, but I know my limitations, and doing good looking UI is one of them.

I had too many software demos blow up because of silly UI issues.

Tags:

Published at

Originally posted at

Comments (11)

An Entity Framework Profiler user story

One of the things that I really love about build things like the profilers and RavenDB is that it gives me the chance to really spread a lot of my experience in codified form. Occasionally, we get customer feedback to that effect, and I thought that I would share this one with you.

Jeremy Holt have been using the Entity Framework Profiler and has the following to say:

The biggest issue it has highlighted for me is the question of the Context being created on different threads (the N+1 was easy to get my head around and resolve). The problem is the difficulty in finding definitive examples of how/when the Context should be created. (Also found my unbound queries – thanks for that!!)

It was working great (except for a couple of threading issues J) until I ran EFProf. I love the “page of shame” in EFProf – I think you called it “query analysis alerts” or something pompous like that – may I suggest you rename it to “errors dumb programmers make” J

I don’t think that this will be renamed, but this does point out something very important. What we are doing with the profilers is much more than simply show you what your application is doing. We are actually analyzing this data through a complex set of rules in order to find common pitfalls. In Jeremy’s case, the common ones were cross thread usage and Select N+1.

And by highlighting those issues, the profiler make it easy to do so. More so, it make you feel good about doing this, in a way that you wouldn’t get from standard analytics tools.

By having thresholds that you can visibly see and avoid, you get to create better software.

Tags:

Published at

Originally posted at

Comments (2)

Your ATM doesn’t use transactions

I just got a series of SMSes from my back, saying that someone just made several withdrawals from my account. As I am currently sitting and watching TV, I was a bit concerned.

It appears that my wife withdrew some money, but there was an issue with one ATM machine, so she used another one.

The problem, I got 3 SMS messages, saying that the follow activities happened on my account:

  • ATM withdrawal for 2,000 NIS
  • ATM withdrawal for 2,000 NIS
  • ATM withdrawal for 2,900 NIS

Checking with my wife, she had actually withdrawn only 2,900.

I was a bit concerned, so I logged into the bank and got this:

image

In English, this is:

Date Description Auth Code Debit Credit
10 Apr ATM withdrawal 00003581 2,000  
10 Apr ATM withdrawal 00003581   2,000
10 Apr ATM withdrawal 00003581 2,900  

This is actually interesting, because the way my wife described it, she wen to the ATM, punch the right codes, and went through the motions of everything. Then, just before it was about to give her the money, it failed.

What is really interesting? From my point of view, is that I can actually see this in my bank account. We didn’t have a transaction rollback because of failure to dispense the money. That isn’t how ATM works. We actually had a compensating action (that occurred as separate transaction) to show that the ATM refunded the money it wasn’t able to give.

So next time someone tries to quote you “banks use transactions”, you can tell them that the bank definition of what a transaction is would make any decent DTC cry with shame.

API Design: Sharding Status for failure scenarios–Solving at the right granularity

This post conclude this week’ series of API design choices regarding how to handle partial failure scenarios in sharded cluster. In my previous post,  I discussed my issues with a local solution for the problem.

The way we ended up solving this issue is actually quite simple. We apply a global solution to a global problem, we added the ability to inject error handling logic deep into the execution pipeline of the sharding implementation, like this:

image

In this case, as you can see, we are allow requests to fail if we are querying (because we can probably still get something from other servers that will be useful), but if you are requesting something by id and it generates an error, we will propagate this error. Note that in our implementation, we call to a user defined “NotifyUserInterfaceAboutServerFailure”, which will let the user know about the error.

That way, you probably have some warning in the UI about partial information, but you are still functional. This is the proper way to handle this, because you are handling this once, and it means that you can handle it properly, instead of having to do the right thing everywhere.

Tags:

Published at

Originally posted at

Comments (15)