Ayende @ Rahien

It's a girl

RavenDB: Splitting entities across several documents

There are occasions where it isn’t feasible or desirable to store our entity as a single document in RavenDB. A question that just came up was how to design votes for an entity using RavenDB.

The scenario is simple, we have our entity, Question (think stack overflow), which can have Up/Down votes. It would be very easy to design the system using a single document for the entity, like so:

{ //document id: questions/123
   Title: "How to handle Up/Down votes with Raven?",
   Content: "...",
   Votes: [
         { Up: true, User: "users/ayende" },
         { Up: false, User: "users/oren" },
  ]
}

As usual, the problem begins when you start to consider what happens when you want to deal with questions that may have large number of votes, or the common scenario where you just want to display the vote totals, and not pull the entire document to get that.

One option is to split things up. I guess you figured that out from the title of this blog post. The idea is to change the document structure to be:

{ //document id: questions/123
   Title: "How to handle Up/Down votes with Raven?",
   Content: "...",
}

{ //document id: questions/123/votes
   Votes: [
         { Up: true, User: "users/ayende" },
         { Up: false, User: "users/oren" },
  ]
}

Note that we have two separate documents here. Now we can load just the questions, or the questions and the votes. We still have a problem with getting the totals without loading potentially thousands of votes. It is pretty easy to solve this, however, using the following index:

from voteDoc in docs.VoteDocs
from vote in voteDoc.Votes
group vote by vote.Up into g
select new { Up = g.Key, Count = g.Count() }

Now we can query the index directly, to get the aggregated results:

session.LuceneQuery<VoteTotals>("Questions/VoteTotals")
            .SelectFields("__document_id", "Up", "Count")
            .ToList();

And if we want to get the votes themselves, they are easily available as well.

CreateSequetialUuid – Answer

I got a lot of interesting answers for the riddle, and here is my solution:

private static int sequentialUuidCounter;
public static Guid CreateSequentialUuid()
{
    var ticksAsBytes = BitConverter.GetBytes(DateTime.Now.Ticks);
    Array.Reverse(ticksAsBytes);
    var increment = Interlocked.Increment(ref sequentialUuidCounter);
    var currentAsBytes = BitConverter.GetBytes(increment);
    Array.Reverse(currentAsBytes);
    var bytes = new byte[16];
    Array.Copy(ticksAsBytes, 0, bytes, 0, ticksAsBytes.Length);
    Array.Copy(currentAsBytes, 0, bytes, 12, currentAsBytes.Length);
    return bytes.TransfromToGuidWithProperSorting();
}

Basically, we use the current system ticks as the 1 – 8 bytes, and a counter incremented atomically on the 12 – 16 bytes. This ensures that even concurrent calls on the same tick will have a different value.

Note that this code explicitly allows the same guid on multiple machines. You can fix that by adding the MAC address as 9 – 12 bytes, which will make this globally unique, but this isn’t something that I actually need.

Additional challenge, why am I reversing the bytes?

Profiler new features: Data binding alerts

The following features apply to NHProf, EFProf, L2SProf.

In general, it is strong discouraged to data bind directly to an IQueryable. Mostly, that is because data binding may actually iterate over the IQueryable several times, resulting in multiple queries being generated from something that can be done purely in memory. Worse, it is actually pretty common for data binding to result in lazy loading, and lazy loading from data binding almost always result in SELECT N+1. The profiler can now detect and warn you about such mistakes preemptively. More than that, the profiler can also now detect queries that are being generated from the views in an ASP.Net MVC application, another bad practice that I don’t like.

You can find more information about each warnings here:

WPF detection:

image

 

image

WinForms detections:

image

image

Web applications:

image

image

Silverlight SEO

One of the reasons that people tell you to avoid Silverlight is the SEO cost. Since Justin’s blog is written in Silverlight, I thought that I might take it for a spin in Lynx, the text only browser, and see what Google sees.

My blog in Lynx:

image_thumb

Justin’s blog is Lynx. His actually looks better than mine in Lynx :-).

image_thumb[1]

Entity Framework: If you pass a connection to the context, you are responsible for disposing it

I recently got a bug report about an issue with EF Prof, apparently the application statistics would record the context being opened, but it wouldn’t show them being closed. The customer was rightfully worried about that, and wanted to know if this is a bug in his code or in EF Prof.

Not closing connections is a pretty bad idea, obviously, because you are going to hold a lot more server resources than you need.

But we couldn’t figure out what the problem was. On my end, I could see that EF Prof was recording the context close properly, but nearly the same code on the customer showed the problem. Note the word nearly. I asked for a repro of the issue, and once I had it, it took mere minutes to confirm that the problem exists.

Now was the time to find out why. The customer code was:

using(var context = new MyContext(new EntityConnection("name=MyConStr"))
{
   // do stuff
}

Now, what EF Prof tells you is a data context open & close are actually more accurately connection open & close. After some extensive study, I verified that it wasn’t my code to blame, there was a leaking connection here.

Checking a bit further, it became clear. We passed an existing connection to the context. When we dispose the context, the context asks: “Am I the owner of this connection?” And since the answer is no, it will not dispose it. It makes sense, you might want to use that connection for your own purposes, and it is pretty rude of the context to close a connection that it doesn’t own.

How can we resolve this? By giving the context the information to open the connection, but not the connection itself:

using(var context = new MyContext("name=MyConStr")
{
   // do stuff
}

This code will instruct the context to open its own connection, which it will know that it owns, so it can safely dispose it when it is disposed.

Look Ma, no leaks!

Tags:

Published at

Originally posted at

Comments (5)

How to find the stack trace for databinding? Answer

This is a rather special request, I’ll admit. For reasons of my own, I need to find the method(s) called to actually performed data binding in WPF & WinForms.

WPF is actually quite simple:

image

WinForms, not so much. I resorted to creating a strongly typed data source and concating my extension there.

image

The answers are, by the way:

WPF:

  • at MS.Internal.Data.DataBindEngine.Task.Run(Boolean lastChance)
  • at MS.Internal.Data.DataBindEngine.Run(Object arg)

WinForms:

  • at System.Windows.Forms.CurrencyManager.SetDataSource(Object dataSource)
  • at System.Windows.Forms.BindingContext.EnsureListManager(Object dataSource, String dataMember)

Published at

How to find the stack trace for data binding?

This is a rather special request, I’ll admit. For reasons of my own, I need to find the method(s) called to actually performed data binding in WPF & WinForms.

I have an elegant solution, which is too large to include in this blog post, but I’ll say that it doesn’t involves reflector at all.

Can you figure it out?

RavenDB: Replicating to a relational database

I just finished implementing a very cool feature for RavenDB, the Index Replication bundle allows you to replicate an index to a relational database.

What does this mean? Well, consider the following document:

var q = new Question
 {
     Title = "How to replicate to SQL Server?",
     Votes = new[]
     {
         new Vote{ Up = true, Comment = "Good!"}, 
         new Vote{ Up = false, Comment = "Nah!"}, 
         new Vote{ Up = true, Comment = "Nice..."}, 
     }
 };

And this index:

from q in docs.Questions
select new 
{
         Title = q.Title, 
         VoteCount = q.Votes.Count
}

With the aid of the Index Replication bundle, that index will be replicated to a relational database, giving us:

image

You can find full documentation for this feature here and the bundle itself is part of RavenDB’s unstable as of build 159.

WPF, Default Buttons and input

I got the following bug report about NH Prof:

When renaming a session, the "enter" key seems to invoke the "Cancel" rather than the "Rename Session" button - this is slightly annoying (and unexpected) behavior.

image 

But that is impossible, here is the XAML for this:

image

And as it turned out, the code is correct, and the explanation for the error is wrong, but the behavior exists.

The actual problem turned out to be that if you hit enter while the focus is on the text box, the RenameSession method will be called, but the bound value will not be modified, since the default binding mode for text boxes is on focus lost. We can fix this using:

image 

Although this is logical, considering how WPF works, I still find it surprising, I would expect this to be a common scenario that would be handled OOTB.

Tags:

Published at

Originally posted at

Comments (9)

Generating API Documentation

I want to generate API documentation for Raven DB. I decided to take the plunge and generate XML documentation, and to ensure that I’ll be good, I marked the “Warnings as Errors” checkbox.

864 compiler errors later (I spiked this with a single project), I had a compiling build and a pretty good initial XML documentation. The next step was to decide how to go from simple XML documentation to a human readable documentation.

Almost by default, I checked Sandcastle, unfortunately, I couldn’t figure out how to get this working at all. Sandcastle Help File Builder seems the way to go, though. And using that GUI, I had a working project in a minute or two. There are a lot of options, but I just went with the default ones and hit build. While it was building, I had the following thoughts:

  • I didn’t like that it required installation.
  • I didn’t know whatever you can easily put it in a build script.

The problem was that generating the documentation (a CHM) took over three and a half minutes.

I then tried Docu. It didn’t work, saying that my assembly was in the wrong format. Took me a while to figure out exactly why, but eventually it dawned on me that Raven is a 4.0 project, while Docu is a 3.5. I tried recompiling it for 4.0, but it gave me an error there as well, something about duplicate mscorlib, at which point I decided to drop it. That was also because I didn’t really like the format of the API it generated, but that is beside the point.

I then tried doxygen, which I have used in the past. The problem here is the sheer number of options you have. Luckily, it is very simple to setup using Doxywizard, and the generated documentation looks pretty nice as well. It also take a while, but significantly faster than Sandcastle.

Next on the list was MonoDoc, where I could generate the initial set of XML files, but was unable to run the actual mono doc browser. I am not quite sure why that happened, but it kept complaining that the result was too large.

I also checked VSDocman, which is pretty nice.

All in all, I think that I’ll go with doxygen, it being the simplest by far and generating (OOTB) HTML format that I really like.

On Silverlight

I read Davy Bryon’s post about the future direction of Silverlight with a great deal of interest. But I think that Davy is missing a significant point.

First, I agree with Davy about a few things. Silverlight for public facing applications is going to be… problematic. Support for mobile devices, something that becomes ever so increasingly important, isn’t really there. And there seems to be a lot more reticence to “bet the farm” on public facing web sites by most companies. A good example of public facing Silverlight application is Justin’s blog. Which is really nice, but has the following issues:

  • It has a long loading time compared to most websites.
  • The scrolling inside the Silverlight application feels… wrong, compared to the one in the browser. It works, but I think it uses a different scroll size than the browser.
  • I can’t zoom with the keyboard (Ctrl –, Ctrl +, Ctrl 0).

All of that doesn’t really matter, to be perfectly frank. Silverlight isn’t really meant for websites, and blogs are probably one of the most common examples of web sites. Silverlight main purpose is applications. Cast your mind a few years back, to the rise of the Web Applications. Why did it happen? Mostly because the cost of deploying desktop software to all the machines in the organization was so high. Building a web application reduced deployment costs.

But web applications are still pretty hard to do, compared to desktop applications, if you want to have similar UX. And the number one problem that keeps recurring is that you have to manually manage the state. Silverlight gives you the same null cost of deployment, but with a lot of the advantages of building desktop applications. Mostly, again, because of the state. Yes, I am well aware that there is a large body of knowledge on how to build complex web applications in pure HTML/JS. But that is still harder than building a Silverlight application.

And that is where I see Silverlight being used most often. It isn’t replacing the company’s site, it is replacing all the internal applications and systems that used to be HTML applications.

Resolving cross site scripting issues.

I got a bug report about the following in the admin UI for RavenDB.

image

As you can imagine, this is certainly something that we would like to avoid, but there is a problem. How the hell do you find the problem?

I mean, obviously we are encoding the value when we present it to the user, since I can see it on the UI. But it is still running, so I am doing something bad here. But I don’t feel like traversing through a mountain of JavaScript to find out exactly where this is happening. Luckily, we don’t have to, we can use the XSS itself to help it localize it:

image

And given that, we can get directly to the actual fault:

image

And fixing that is a snap.

Profiler new features, Sept Edition

The following features apply to NHProf, EFProf, HProf, L2SProf.

The first feature is something that was frequently requested, but we kept deferring. Not because it was hard, but because it was tedious and we had cooler features to implement: Sorting.

image

Yep. Plain old sorting for all the grids in the application.

Not an exciting feature, I’ll admit, but an important one.

The feature that gets me exciting is the Go To Session. Let us take the Expensive Queries report as a great example for this feature:

image

As you can see, we have a very expensive query. Let us ignore the reason it is expensive, and assume that we aren’t sure about that.

The problem with the reports feature in the profiler is that while it exposes a lot of information (expensive queries, most common queries, etc), it also lose the context of where this query is running. That is why you can, in any of the reports, right click on a statement and go directly to the session where it originated from:

image

image

We bring the context back to the intelligence that we provide.

What happen if we have a statement that appear in several sessions?

image

You can select each session that this statement appears in, getting back the context of the statement and finding out a lot more about it.

I am very happy about this feature, because I think that it closes a circle with regards to the reports. The reports allows you to pull out a lot of data across you entire application, and the Go To Session feature allows you to connect the interesting pieces of the data back to originating session, giving you where and why this statement was issued.

Can you make this code run any faster?

public static class GuidExtensions
{
    public static Guid TransfromToGuidWithProperSorting(this byte[] bytes)
    {
        return new Guid(new[]
        {
            bytes[10],
            bytes[11],
            bytes[12],
            bytes[13],
            bytes[14],
            bytes[15],
            bytes[8],
            bytes[9],
            bytes[6],
            bytes[7],
            bytes[4],
            bytes[5],
            bytes[0],
            bytes[1],
            bytes[2],
            bytes[3],
        });
    }

    public static byte[] TransformToValueForEsentSorting(this Guid guid)
    {
        var bytes = guid.ToByteArray();
        return new[]
        {
            bytes[12],
            bytes[13],
            bytes[14],
            bytes[15],
            bytes[10],
            bytes[11],
            bytes[8],
            bytes[9],
            bytes[6],
            bytes[7],
            bytes[0],
            bytes[1],
            bytes[2],
            bytes[3],
            bytes[4],
            bytes[5],
        };
    }
}

 

Just to note, this is stupid micro optimization trick, it takes 0.00008 ms to execute right now, which is plenty fast enough, but it is fun to play with it.

How to reproduce an occasionally failing test?

One of the worst possible things that can happen is a test that fails, sometimes. Not always, and never under the debugger, but it fails.

It tells you that there is a bug, but doesn’t give you the tool to find it:

image

Usually, this is an indication of a problem with the code that is exposed through multi threading. I found the following approach pretty useful in digging those bastards out:

static void Main()
{
    for (int i = 0; i < 100; i++)
    {
        using (var x = new Raven.Tests.Indexes.QueryingOnDefaultIndex())
        {
            x.CanPageOverDefaultIndex();
            Console.Write(".");
        }
    }
}

Yes, it is trivial, but you would be surprised how often I see people throwing their hands in despair over issues like this.

It ain’t no simple feature, mister

I recently got a bug report about NH Prof in a multi monitor environment. Now, I know that NH Prof works well in multi monitor environment, because I frequently run in such an environment myself.

The problem turned out to be not multi monitors in and of itself, but rather how NH Prof handles the removal of a monitor. It turns out that NH Prof has a nice little feature that actually remembers the last position the window was at, and returns to it on start. When the monitor NH Prof was located on was removed, on start NH Prof would put itself beyond the screen position.

That led to me having to figure out how to find the available monitor space, so I could detect if the saved positions were valid or not. What I found interesting in this is that what seemed to be a very trivial feature (save two numbers) turned out to be somewhat more complex, and I am pretty sure that there are other scenarios that I am missing (in the very same feature).

Implementing CreateSequentialUuid()

We run into an annoying problem in Raven regarding the generation of sequential guids. Those are used internally to represent the etag of a document.

For a while, we used the Win32 method CreateSequentialUuid() to generate that. But we run into a severe issue with that, it create sequential guids only as long as the machine is up. After a reboot, the guids are no longer sequential. That is bad, but it also means that two systems calling this API can get drastically different results (duh! that is the point, pretty much, isn’t it?). Which wouldn’t bother me, except that we use etags to calculate the freshness of an index, so we have to have an always incrementing number.

How would you implement this method?

public static Guid CreateSequentialUuid()

A few things to note:

  • We really actually care about uniqueness here, but only inside a single process, not globally.
  • The results must always be incrementing.
  • The always incrementing must be consistent across machine restarts and between different machines.

Yes, I am fully aware of the NHibernate’s implementation of guid.comb that creates sequential guids. It isn't applicable here, since it doesn't create truly sequential guids, only guids that sort near one another.

Estimates sucks, especially when I pay for them

I recently got an estimate for a feature that I wanted to add to NH Prof. It was for two separate features, actually, but they were closely related.

That estimate was for 32 hours.

And it caused me a great deal of indigestion. The problem was, quite simply, that even granting that there is the usual padding of estimates (which I expect), that timing estimate was off, way off. I knew what would be required for this feature, and it shouldn’t be anywhere near complicated enough to require 4 days of full time work. In fact, I estimated that it would take me a maximum of 6 hours and a probable of 3 hours to get it done.

Now, to be fair, I know the codebase (well, actually that isn’t true, a lot of the code for NH Prof was written by Rob & Christopher, and after a few days of working with them, I stopped looking at the code, there wasn’t any need to do so). And I am well aware that most people consider me to be an above average developer.

I wouldn’t have batted an eye for an estimate of 8 – 14 hours, probably. Part of the reason that I have other people working on the code base is that even though I can do certain things faster, I can only do so many things, after all.

But a time estimate that was 5 – 10 times as large as what I estimated was too annoying. I decided that this feature I am going to do on my own. And I decided that I wanted to do this on the clock.

The result is here:

image

This is actually total time over three separate sittings, but the timing is close enough to what I though it would be.

This includes everything, implementing the feature, unit testing it, wiring it up in the UI, etc.

The only thing remaining is to add the UI works for the other profilers (Entity Framework, Linq to SQL, Hibernate and the upcoming LLBLGen Profiler) . Doing this now…

image

And we are done.

I have more features that I want to implement, but in general, if I pushed those changes out, they would be a new release that customers can use immediately.

Nitpicker corner: No, I am not being ripped off. And no, the people making the estimates aren't incompetent. To be perfectly honest, looking at the work that they did do and the time marked against it, they are good, and they deliver in a reasonable time frame. What I think is going on is that their estimates are highly conservative, because they don't want to get into a bind with "oh, we run into a problem with XYZ and overrun the time for the feature by 150%".

That also lead to a different problem, when you pay by the hour, you really want to have estimates that are more or less reasonably tied to your payments. But estimating with hours has too much granularity to be really effective (a single bug can easily consume hours, totally throwing off estimation, and it doesn't even have to be a complex one.)

Moving away from the Active Record model

Fowler defines Active Record as:

An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.

Note that I am talking about the Active Record pattern and not a particular Active Record implementation.

A typical Active Record class will look like this:

image

By design, AR entities suffer from one major problem, they violate SRP, since the class is now responsible for its own persistence and for the business operations that relates to it. But that is an acceptable decision in many cases, because it leads to code that is much simpler to understand.

In most cases, that is.

The problem with AR is that it also treat each individual entity independently. In other words, its usage encourage code like this:

var person = Person.FindBy("Id", 1);
person.Name = "Oren";
person.Save();

This is a very procedural way of going about things, but more importantly, it is a way that focus on work done on a single entity.

When I compare the abilities of the Active Record style to the abilities of a modern ORM, there is really nothing that can compare. Things like automatic change tracking or batching writes to the database are part of the package with an ORM.

The sort of applications we write today aren’t touching a single row in the database, we tend to work with entities (especially root aggregates) with operations that can touch many rows. Writing those sort of applications using Active Record leads to complications, precisely because the pattern is meant to simplify data access. The problem is that our data access needs aren’t so simple anymore.

Active Record can still be useful for very simple scenarios, but I think that a major drive behind it was that it was so easy to implement compared to a more full featured solution. Since we don’t need to implement a full featured solution (it comes for free with your ORM of choice), that isn’t really a factor anymore. Using a full blown ORM is likely to be easier and simpler.

Maintainability, Code Size & Code Complexity

The most maintainable codebase that I worked with grew at a rate of about ~10 KLoC per month, every month. There wasn’t a large team there, it ranged fro 3 – 6 people. This is the project that I think about whenever I had to talk about good code bases.

It is a WebForms project (under protest, but it is).

What make it maintainable? Not the WebForms part, which wouldn’t come as a surprise. What make it maintainable is that it is the first project where I had applied the notion that large swaths of simple code is better than smaller code base with higher complexity. You could see example of this sort of architecture in the Alexandria and Effectus applications.

Note: I am doing things like measure KLOC here mostly because it is a number that I can measure. As I am saying in this post, KLOC has very little to do with maintainability.

The problem can be expressed well using the graphs. This is a representation of the complexity of the application as it grows, lower maintainability cost is better.

image

The problem with the smaller and more complex code base is that the complexity tends to explode very quickly. With the larger code base, you end up more or less on the same plane.

But the above chart is somewhat misleading, because it make the hidden assumption that in both code styles, you’ll have the same amount of code, which is obviously false. Here is what is probably a more accurate representation of how much code is written for style per # of features.

 

image

This doesn’t look right. Not when we compare it to the chart above. Of the top of my head, I would expect the second chart to be the mirror image of the first one. But it isn’t. And the reason that it isn’t is that each feature you write still cost you some amount of code. And the rate of growth per features added is pretty constant either way.

Putting the two charts together, you can see that it means that even code styles with focus on less code in favor of more complex solutions grow, and that as they grow, they become less maintainable.

So far I have been very vague when I was talking about “complex” and “simple”. In part, this is because those are somewhat hard to define. There would be people who would claim that ORM leads to complex codebases, but I would obviously disagree.

For my part, I also strongly advocate of having a strong core of infrastructure that gives services for the rest of the application, and that tend to be complex piece of coding. But that is also something that is fixed, once the infrastructure is written, it tend to be static, so that saves you from the fate outlined above.

When I look at a piece of code, I do the usual evals (nesting, conditionals, cyclomatic complexity, etc), but I also look at how often the developers reached for a hammer and beat everything around to submission. There is a lot of gut feeling here. But I do have one objective metric to use to tell you whatever a piece of code fit either style.

In the fight between Don’t Repeat Yourself an Single Responsibility Principle, SRP wins, every single time.

Any time that I have seem code that did double (or triple or dozenile) duty, it led to the sort of “we have to write less code” style of coding that ended up in a quagmire.

Tags:

Published at

Originally posted at

Comments (39)

RavenDB performance optimizations

Just to note, you’ll probably read this post about a month after the change was actually committed.

I spent the day working on a very simple task, reducing the number of writes that RavenDB makes when we perform a PUT operation. I managed to reduce one write operation from the process, but it took a lot of work.

I thought that I might show you what removing a single write operation means, so I built a simple test harness to give me consistent numbers (in the source, look for Raven.Performance).

Please note that the perf numbers are for vanilla RavenDB, with the default configuration, running in debug mode. We can do better than that, but what I am interested in is not absolute numbers, but the change in those numbers.

Here are the results for build 124, before the change:

Wrote 5,163 documents in 5,134ms: 1.01: docs/ms
Finished indexing in 8,032ms after last document write

And here are the numbers for build 126, after the change:

Wrote 5,163 documents in 2,559ms: 2.02: docs/ms
Finished indexing in 2,697ms after last document write

So we get double the speed at write time, but we also get much better indexing speed, this is sort of an accidental by product, because now we index documents based on range, rather than on specific key. But it is a very pleasant accident.

Tags:

Published at

Originally posted at

Comments (15)

Being a product owner

A while ago I have come to the realization that it is impossible for me to everything alone, so I got other people to help me build my projects. In some cases, that was done using OSS, by soliciting contribution from the community. In others, it is a simple commercial transaction, where I give someone money for code.

I think that I have gotten too used to the OSS model, because I got the following reply for this:

image

That was somewhat of a rude shock, I was using the same loose language that I always hated when I got specs to implement.

Those are all really good questions.