Ayende @ Rahien

I think that I’ll ignore this test failure

Thu, 24 May 2012 09:00:00 GMT

This really sucks, because of this:

This is part of the RavenDB release process, where we are actually hammering RavenDB and seeing if it breaks. And it usually takes about 24 hours to do a complete run.

The RavenDB Release Process

Wed, 23 May 2012 09:00:00 GMT

We got several questions about this in the mailing list, so I thought that this would be a good time to discuss this in the blog.

One of the best part about RavenDB is that we are able to deliver quickly and continuously. That means that we can deliver changes to the users very rapidly, often resulting in response times of less than an hour from “I have an issue” to “it is already fixed and you can download it”.

That is awesome on a lot of level, but it lack something very important, stability. In other words, by pushing things so rapidly, we are living on the bleeding edge. Which is great, except that you tend to bleed.

That is why we split the RavenDB release process into Unstable and Stable. Unstable builds are released on a “several times a day” basis, and only require that we will pass our internal test suite. This test suite is hefty, over 1,500 tests so far, but it is something that can be run in about 15 minutes or so on the developer machine to make sure that our changes didn’t break anything.

The release process for Stable version is much more involved. First, of course, we run the standard suite of tests. Then we have a separate set of tests, which are stress testing RavenDB by trying to see if there are any concurrency issues.

Next, we take the current bits and push them to our own internal production systems. For example, at the time of this writing, this blog (and all of our assets) are currently running on RavenDB build 726 and have been running that way for a few days. This allows us to test several things. That there are no breaking changes, that this build can survive running in production over extended period of time and that the overall performance remains excellent.

Finally, we ask users to take those builds for a spin, and they are usually far more rough on RavenDB than we are.

After all of that, we move to a set of performance tests, comparing the system behavior on a wide range of operations compared to the old version.

And then… we can do a stable release push. Phew!

RavenHQ goes out of beta

Tue, 22 May 2012 07:00:00 GMT

After several months in public beta, I am proud to announce that RavenHQ, the RavenDB as a Service on the cloud has dropped the beta label and is now ready for full production use.

That means that we now accept signups from the general public, you no longer need an AppHarbor account and you can use it directly. It also means that you can safely start using RavenHQ for production purposes.

RavenHQ is a fully-managed cloud of RavenDB servers and scalable plans, you’ll never have to worry about installation, updates, availability, performance, security or backups again.

We offer both standard and high availability plans, and are the perfect fit for RavenDB users who can safely outsource all the operational support of your databases in the RavenHQ’s team capable hands.

Reviewing RavenDB app: ReleaseCandidateTracker

Wed, 16 May 2012 09:00:00 GMT

ReleaseCandidateTracker is a new RavenDB based application by Szymon Pobiega. I reviewed version 5f7e42e0fb1dea70e53bace63f3e18d95d2a62dd. At this point, I don’t know anything about this application, including what exactly it means, Release Tracking.

I downloaded the code and started VS, there is one project in the solution, which is already a Good Thing. I decided to randomize my review approach and go and check the Models directory first.

Here is how it looks:

This is interesting for several reasons. First, it looks like it is meant to keep a record of all deployments to multiple environments, and that you can lookup the history of each deployment both on the environment side and on the release candidate side.

Note that we use rich models, which have collections in them. In fact, take a look at this method:

Which calls to this method:

You know what the really fun part about this?

It ain’t relational model. There is no cost of actually making all of these calls!

Next, we move to the Infrastructure folder, where we have a couple of action results and the RavenDB management stuff. Here it how RCT uses RavenDB:

public static class Database
{
    private static IDocumentStore storeInstance;

    public static IDocumentStore Instance
    {
        get
        {
            if (storeInstance == null)
            {
                throw new InvalidOperationException("Document store has not been initialized.");
            }
            return storeInstance;
        }
    }

    public static void Initialize()
    {
        var embeddableDocumentStore = new EmbeddableDocumentStore {DataDirectory = @"~\App_Data\Database"};
        embeddableDocumentStore.Initialize();
        storeInstance = embeddableDocumentStore;
    }
}

It is using an embedded database to do that, which makes it very easy to use the app. Just hit F5 and go. In fact, if we do, we see the fully functional website, which is quite awesome .

Let us move to seeing how we are managing the sessions:

public class BaseController : Controller
{
    public IDocumentSession DocumentSession { get; private set; }
    public CandidateService CandidateService { get; private set; }
    public ScriptService ScriptService { get; private set; }

    protected override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        if (filterContext.IsChildAction)
        {
            return;
        }
        DocumentSession = Database.Instance.OpenSession();
        CandidateService = new CandidateService(DocumentSession);
        ScriptService = new ScriptService(DocumentSession);
        base.OnActionExecuting(filterContext);
    }
    
    protected override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        if (filterContext.IsChildAction)
        {
            return;
        }
        if(DocumentSession != null)
        {
            if (filterContext.Exception == null)
            {
                DocumentSession.SaveChanges();
            }
            DocumentSession.Dispose();
        }
        base.OnActionExecuted(filterContext);
    }
}

This is all handled inside the base controller, and it is very similar to how I am doing that in my own apps.

However, ScriptService and CandidateService seems strange, let us explore them a bit.

public class ScriptService
{
    private readonly IDocumentSession documentSession;

    public ScriptService(IDocumentSession documentSession)
    {
        this.documentSession = documentSession;
    }

    public void AttachScript(string versionNumber, Stream fileContents)
    {
        var metadata = new RavenJObject();
        documentSession.Advanced.DatabaseCommands.PutAttachment(versionNumber, null, fileContents, metadata);
    }

    public Stream GetScript(string versionNumber)
    {
        var attachment = documentSession.Advanced.DatabaseCommands.GetAttachment(versionNumber);
        return attachment != null 
            ? attachment.Data() 
            : null;
    }
}

So this is using RavenDB attachment to store stuff, I am not quite sure what yet, so let us track it down.

This is being used like this:

[HttpGet]
public ActionResult GetScript(string versionNumber)
{
    var candidate = CandidateService.FindOneByVersionNumber(versionNumber);
    var attachment = ScriptService.GetScript(versionNumber);
    if (attachment != null)
    {
        var result = new FileStreamResult(attachment, "text/plain");
        var version = candidate.VersionNumber;
        var product = candidate.ProductName;
        result.FileDownloadName = string.Format("deploy-{0}-{1}.ps1", product, version);
        return result;
    }
    return new HttpNotFoundResult("Deployment script missing.");
}

So I am assuming that the scripts are deployment scripts for different versions, and that they get uploaded on every new release candidate.

But look at the CandidateService, it looks like a traditional service wrapping RavenDB, and I have spoken against it multiple times.

In particular, I dislike this bit of code:

public ReleaseCandidate FindOneByVersionNumber(string versionNumber)
{
    var result = documentSession.Query<ReleaseCandidate>()
        .Where(x => x.VersionNumber == versionNumber)
        .FirstOrDefault();
    if(result == null)
    {
        throw new ReleaseCandidateNotFoundException(versionNumber);
    }
    return result;
}

public void Store(ReleaseCandidate candidate)
{
    var existing = documentSession.Query<ReleaseCandidate>()
        .Where(x => x.VersionNumber == candidate.VersionNumber)
        .Any();
    if (existing)
    {
        throw new ReleaseCandidateAlreadyExistsException(candidate.VersionNumber);
    }
    documentSession.Store(candidate);
}

From looking at the code, it looks like the version number of the release candidate is the primary way to look it up. More than that, in the entire codebase, there is never a case where we load a document by id.

When I see a VersionNumber, I think about things like “1.0.812.0”, but I think that in this case the version number is likely to include the product name as well, “RavenDB-1.0.812.0”, otherwise you couldn’t have two products with the same version.

That said, the code above it wrong, because it doesn’t take into account RavenDB’s indexes BASE nature. Instead, the version number should actually be the ReleaseCandidate id. This way, because RavenDB’s document store is fully ACID, we don’t have to worry about index update times, and we can load things very efficiently.

Pretty much all of the rest of the code in the CandidateService is only used in a single location, and I don’t really see a value in it being there.

For example, let us look at this one:

[HttpPost]
public ActionResult MarkAsDeployed(string versionNumber, string environment, bool success)
{
    CandidateService.MarkAsDeployed(versionNumber, environment, success);
    return new EmptyResult();
}

As you can see, it is merely loading the appropriate release candidate, and calling the MarkAsDeployed method on it.

Instead of doing this needless, forwarding, and assuming that we have the VersionNumber as the id, I would write:

[HttpPost]
public ActionResult MarkAsDeployed(string versionNumber, string environment, bool success)
{
    var cadnidate = DocumentSession.Load<ReleaseCandidate>(versionNumber);
    if (cadnidate == null)
        throw new ReleaseCandidateNotFoundException(versionNumber);
    var env = DocumentSession.Load<DeploymentEnvironment>(environment);
    if (env == null)
        throw new InvalidOperationException(string.Format("Environment {0} not found", environment));

    cadnidate.MarkAsDeployed(success, env);
    return new EmptyResult();
}

Finally, a word about the error handling, this is handled via:

protected override void OnException(ExceptionContext filterContext)
{
    filterContext.Result = new ErrorResult(filterContext.Exception.Message);
    filterContext.ExceptionHandled = true;
}

public class ErrorResult : ActionResult
{
    private readonly string message;

    public ErrorResult(string message)
    {
        this.message = message;
    }

    public override void ExecuteResult(ControllerContext context)
    {
        context.HttpContext.Response.Write(message);
        context.HttpContext.Response.StatusCode = 500;
    }
}

The crazy part is that OnException is overridden only on some of the controllers, rather than in the base controller, and even worse. This sort of code leads to error details loss.

For example, let us say that I get a NullReferenceException. This code will dutifully tell me all about it, but will not tell me where it happened.

This sort of thing make debugging extremely hard.

I hate this code

Thu, 03 May 2012 09:00:00 GMT

I really hate this code, it is so stupid it makes my head hurt, and it have so much important factors in it. In particular, there is a lot of logic in here.

You might not see it as such, but a lot of this is actually quite important, default values, config parsing, decisions.

This is important. And it is all handled in a a few methods that goes on forever and hide important details in the tediousness of parameter unpacking.

This approach works if you have 5 parameters, not when you have 50.

And sometimes Things Just Works

Tue, 01 May 2012 09:00:00 GMT

I am in the process of writing an article about RavenDB, and I just wrote the following code to demonstrate RavenDB schema less nature:

using (var session = documentStore.OpenSession())
{
    session.Store(new Customer
    {
        Name = "Joe Smith",
        Attributes =
            {
                {"IsAnnoyingCustomer", true},
                {"SatisfactionLevel", 8.7},
                {"LicensePlate", "B7D-12JA"}
            }
    });

    session.SaveChanges();
}

using (var session = documentStore.OpenSession())
{
    var customers = session.Query<Customer>()
        .Where(x => x.Attributes["IsAnnoyingCustomer"].Equals(true))
        .ToList();

    Console.WriteLine(customers.Count);

    session.SaveChanges();
}

This worked, flawlessly.

The amount of work that we have put into RavenDB to make such things work is really scary when you sit down to think about it.

But it works, it does what I expect it to do and it doesn’t get in my way, woohoo!

Relational searching sucks, don’t try to replicate it

Fri, 27 Apr 2012 09:00:00 GMT

This question on Stack Overflow is a fairly common one. Here is the data:

And the question was about how to get RavenDB to create an index that would have the following results:

{
   CarId: "cars/1",
   PersonId: "people/1235",
   UnitId: "units/4321",
   Make: "Toyota",
   Model: "Prius"
   FirstName: "Ayende",
   LastName: "Rahien"
   Address: "Komba 10, Hadera"
}

{
   CarId: "cars/2",
   PersonId: "people/1236",
   UnitId: "units/4321",
   Make: "Toyota",
   Model: "4runner"
   FirstName: "test",
   LastName: "test"
   Address: "blah blah"
}

same unit different person owns a different car

Now, if you try really hard, you can probably try to get something like that, but that is the wrong way to go about this in RavenDB.

Instead, we can write the following index:

Note that this index is a simple multi map index, it isn’t a multi map/reduce index. There is no need.

This index can return one of three types.

Car – just show the car to the user
Person – now that we have a person, we have the id, and we can query for that:
Unit – now that we have a unit, we have the id, and we can query for that:

This method means that we have to generate an additional query for some cases, but it has a lot of advantages. It is simple. It requires very little work from both client and server and it doesn’t suffer from the usual issues that you run into when you attempt to query over multiple disjointed data sets.

Now, the bad thing about this is that this won’t allow me to query for cross entity values, so it would be hard for me to query for the cars in Hadera owned by Ayende. But in most cases, that isn’t really a requirement. We just want to be able to search by either one of those, not all of them.

As the user’s put it: Insight into the RavenDB design mindset

Thu, 26 Apr 2012 06:00:00 GMT

I have been blogging for a long time now, and I am quite comfortable in expressing myself, but I was still blown away by this post to the RavenDB mailing list. Mostly because this thread sums up a lot of the core points that led me to design RavenDB the way it is today.

Rasmus Schultz has been able to put a lot of the thought processes behind the RavenDB design into words.

Back when I took my education in systems development, basically, I was taught to build aggregates as large, as complete and as connected as possible. But that was 14 years ago, and I'm starting to think, what they taught me back then was based on the kind of thinking that works for single-user, typically desktop applications, where the entire model was assumed to be in-memory, and therefore had to be traversible, since there was no "engine" you could go back to and ask for another piece of the model.
I can see now why that doesn't make sense for concurrent applications with large models persisted in the background. It just never occurred to me, and looked extremely wrong to me, because that's not how I was taught to think.

Yes. That is the exact problem that I see people run into over and over. The create highly connected object model, without regards to how they are persisted, and then they run into problems using them. And the assumption that everything is equally costly to read from memory is hugely expensive.

Furthermore, I'm starting to see why NHibernate doesn't really work well for me. So here's the main thing that's starting to dawn on me, and please confirm or correct me on this:
It seems that the idea behind NH is to configure the expected data-access strategies for the model itself. You write configuration-files that define the expected data-access strategies, but potentially, you're doing this based on assumptions about how you might access the data in this or that scenario.
The problem I'm starting to see, is that you're defining these assumptions statically - and while it is possible to deviate from these defined patterns, it's easy to think that once you've defined your access strategies, you're "done", and the model "just works" and you can focus on writing business logic, which too frequently turns out to be untrue in practice.

To be fair, you can specify those things in place, with full context. And I have been recommending to do just that for years, but yeah, that is a very common issue.

This contrasts with RavenDB, where you formally define the access strategies for specific scenarios - rather than for the model itself. And of course the same access strategy may work in different scenarios, but you're not tempted to assume that a single access strategy is going to work for all scenarios.
You're encouraged to think and make choices about what you're accessing and updating in each scenario, rather than just defining one overriding strategy and charging ahead blindly on the assumption that it'll always just work, or always perform well, or always make updates that are sufficiently small to not cause concurrency problems.
Am I catching on?

Precisely.

Lazy’s Man comprehensive search with RavenDB

Wed, 25 Apr 2012 09:00:00 GMT

RavenDB supports many types of searches, and in this case, I want to show something that belongs to the cool parts of the pile, but also on the “you probably don’t really want to do this”.

First, let me explain why this is cool, then we will talk about why you probably don’t want to do that (and finally, about scenarios where you actually do want this).

Here is an index that will allow you to search over all of the values of all of the properties in the user entity:

public class Users_AllProperties : AbstractIndexCreationTask<User, Users_AllProperties.Result>
{
    public class Result
    {
        public string Query { get; set; }
    }
    public Users_AllProperties()
    {
        Map = users =>
              from user in users
              select new
              {
                  Query = AsDocument(user).Select(x => x.Value)
              };
        Index(x=>x.Query, FieldIndexing.Analyzed);
    }
}

This can be easily query for things like:

s.Query<Users_AllProperties.Result, Users_AllProperties>()
    .Where(x=>x.Query == "Ayende") // search first name
    .As<User>()
    .ToList()


s.Query<Users_AllProperties.Result, Users_AllProperties>()
    .Where(x=>x.Query == "Rahien") // search last name
    .As<User>()
    .ToList()

The fun part is that because we are actually going to index all the properties values into the Query field, which then allow us to easily query for every one of the values without any trouble.

The problem with that is that this is also quite wasteful and likely to lead to bad results down the road. Why?

For two major reasons. First, because this is going to index everything, and would result in larger index, more IO, etc. The second reason is that it is going to lead to bad results because you are now searching over everything, including the “last login date” and the “password hint”. That means that your search results relevancy is going to be poor.

So why would you ever want to do something like that if it is bad?

Well, there are a few scenarios where this is applicable. You need to do that if you want to be able to search over completely / mostly dynamic entities. And you want to do that if you have entities which are specifically generated for the purpose of being searched.

Both cases are fairly rare (the first case is usually covered by dynamic indexing, anyway), so I wanted to point this out, and also point out that it is usually far better to just specify what are the fields that actually matter for you.

The RavenDB indexing process: Optimization–Tuning? Why, we have auto tuning

Tue, 24 Apr 2012 09:00:00 GMT

The final aspect of RavenDB’s x7 jump in indexing performance is the fact that we made it freakishly smart.

During standard operation, most indexes only update when new information comes in, we are usually talking about a small number of documents for every indexing run. The problem is what happens when you have a sudden outpour of documents into RavenDB? For example, during nightly ETL batch, or just if you suddenly have a flood of users doing write operations.

The problem here is that we actually have to balance a lot of variable at the same time:

The number of documents that we have to index*.
The current memory utilization**.
How any cores I have available to do the index work with?
How much time do I have to do this?

Basically, the idea goes like this, if I have a small batch size, I am able to index more quickly, ensuring that we have fresher results. If I have big batch size, I am able to index more documents, and my overall indexing times goes down.

There is a non trivial cost associated with every indexing run, so reducing the number of indexing run is good, but the more documents I shove into a single run, the more memory will I use, and the more time it will take before the results are visible to the users.

* It is non trivial because there is no easy way for us to even know how many documents we have left to index (to find out is costly).

** Memory utilization is hard to figure out in a managed world. I don’t actually have a way to know how much memory I am using for indexing and how much for other stuff, and there is no real way to say “free the memory from the last indexing run”, or even estimate how much memory that took.

What we have decided on doing is to start from a very small (low hundreds) indexing batch size, and see what is actually going on live. If we see that we have more documents to index than the current batch size, we will slowly double the size of the batch. Slowly, because bigger batches requires more memory, and we also have to take into account current utilization, memory usage, and a bunch of other factors as well. We also go the other way around, able to reduce the indexing batch size on demand based on how much work we have to do right now.

We also provide an upper limit, because at some point it make sense to just do a big batch and make the indexing results visible than to try to do everything all at once.

The fun part in all of that is that once we have found the appropriate algorithm for this, it means that RavenDB will automatically adjust itself based on real production load. If you have an low update rate, it will favor small indexing batches and immediately execute indexing on the new documents. However, if you suddenly have a spike in traffic and the update rate goes up, RavenDB will adjust the indexing batch size so it will be able to keep up with your rate.

We have done some (read, a huge amount) testing with regards to this new optimization, and it turns out that under slow update frequency, we are seeing an average of 15 – 25 ms between a document update and it showing up in the indexes. That is pretty good, but what is going on when we have data just pouring in?

We tested this with a 3 million documents and 3 indexes. And it turn out that under this scenario, where we are trying to shove data into RavenDB as fast as it can accept it, we do see an increase in index latency. Under those condition, latency rose all the way to 1.5 seconds.

This is actually something that I am very happy about, because we were able to automatically adjust to the changing conditions, and were still able to index things at a reasonable rate (note that under this scenario, the batch size was usually 8 – 16 thousands documents, vs. the 128 – 256 that it is normally).

Because we were able to adjust the batch size on the fly, we could handle sustained writes at this rate with no interruption in service and no real need to think about this from the users perspective.. Exactly what the RavenDB philosophy calls for.

The RavenDB indexing process: Optimization–Getting documents from disk

Mon, 23 Apr 2012 07:00:00 GMT

As I noted in my previous post, we have done major optimizations for RavenDB. One of the areas where we improved the performance was reading the documents from the disk for indexing.

In Pseudo Code, it looks like this:

while database_is_running:
  stale = find_stale_indexes()
  lastIndexedEtag = find_last_indexed_etag(stale)
  docs_to_index = get_documents_since(lastIndexedEtag, batch_size)
  

As it turned out, we had a major optimization option here, because of the way the data is actually structured on disk. In simple terms, we have an on disk index that lists the documents in the order in which they were updated, and then we have the actual documents themselves, which may be anywhere on the disk.

Instead of loading the documents in the orders in which they were modified, we decided to try something different. We first query the information we need to find the document on disk from the index, then we sort them based on the optimal access pattern, to reduce disk movement and ensure that we have as sequential reads as possible. Then we take those results in memory and sort them based on their last update time again.

This seems to be a perfectly obvious thing to do, assuming that you are aware of such things, but it is actually something that is very easy not to notice. The end result is quite promising, and it contributed to the 7+ times improvements in perf that we had for indexing costs.

But surprisingly, it wasn’t the major factor, I’ll discuss a huge perf boost in this area tomorrow.

RavenDB 1.2 work has started (and a road map)

Sun, 22 Apr 2012 09:43:00 GMT

Two years after the launch of RavenDB 1.0, (preceded by several years of working on 1.0, of course). We are now starting to actually plan and work on RavenDB 1.2.

You can read the planned roadmap here. RavenDB 1.2 is a big release, for several reasons.

We are going to break RavenDB into several distinct editions, from the RavenDB Basic, suitable for small apps to RavenDB Standard which is the current version and all the way up to RavenDB enterprise, which is going to get some awesome features (windows clustering, index encryption, etc). We are also going to have plans for ISVs, which will allow them royalty free distribution of RavenDB for their customers.
We are going to update our pricing structure. You’ll hear more about this when we have finalized pricing.

Because I am well aware of the possible questions, I suggest reading the thread discussing both editions and pricing in the mailing list:

I will repeat again that we haven’t yet made final pricing decisions, so don’t take the numbers thrown around in those threads as gospel, but they are pretty close to what we will have.

This is the boring commercial stuff, but I am much more interested in talking about the new RavenDB roadmap. In fact, you can actually read all of our plans here. The major components for RavenDB 1.2 are:

Better integration with C# 5.0 – much better support for async in general, async replicaiton, async sharding, etc.
Enterprise level features – Windows Clustering, Full Database Encryption, Indexing Priorities, Compression, etc.
Installer and server console - so you can manage your RavenDB installation more easily.
Better Admin support – scheduled backups, S3 Backups, live restores, etc.
Internalizing commonly used bundles – you shouldn’t have to take additional steps to make use of common functionality.

There are other stuff, of course, but those are the main pillars.

As mentioned, you can read all of that yourself, and we would welcome feedback on our current plans and suggestions for the new version.

The RavenDB indexing process: Optimization–De-parallelizing work

Fri, 20 Apr 2012 09:00:00 GMT

One of the major dangers in doing perf work is that you have a scenario, and you optimize the hell out of that scenario. It is actually pretty easy to do without even noticing it. The problem is that when you do things like that, you are likely to be optimizing a single scenario to perform really well, but you are hurting the overall system performance.

In this example, we have moved heaven and earth to make sure that we are indexing things as fast as possible, and we tested with 3 indexes, on an 4 cores machine. As it turned out, we actually had improved things, for that particular scenario.

Using the same test case on a single core machine was suddenly far more heavy weight, because we were pushing a lot of work at the same time. More than the machine could process. The end result was that it actually got there, but much more slowly than if we would have run things sequentially.

Of course, I give you the outliers, but those are good indicators for what we found out. Initially, we thought that we could resolve that by using the TPL’s MaxDegreeOfParallelism, but it turned out to be more complex than that. We have IO bound and we have CPU bound tasks that we need to execute, and trying to execute IO heavy tasks with this would actually cause issues in this scenario.

We had to manually throttle things ourselves, both to ensure limited number of parallel work, and because we have a lot more information about the actual tasks than the TPL have. We can schedule them in a way that is far more efficient because we can tell what is actually going on.

The end result is that we are actually using less parallelism, overall, but in a more efficient manner.

In my next post, I’ll discuss the auto batch tuning support, which allows us to do some really amazing things from the point of view of system performance.

The RavenDB indexing process: Optimization–Parallelizing work

Thu, 19 Apr 2012 09:00:00 GMT

One of the things that we are doing during the index process for RavenDB is applying triggers and deciding what, if and how a document will be indexed. The actual process is a bit more involved, because we have to do additional things (like figure out which indexes have already indexed those particular documents).

At any rate, the interesting thing is that this is a process which is pretty basic:

for doc in docs:
    matchingIndexes = FindIndexesFor(doc)
    if matchingIndexes.Count > 0:
       doc = ExecuteTriggers(doc) 
       if doc != null:
          yield doc

The interesting thing about this is that this is a set of operations that only works on a single document at a time, and the result is the modified documents.

We were able to gain significant perf boost by simply moving to a Parallel.ForEach call. This seems simple enough, right? Parallelize the work, get better benefits.

Except that there are issues with this as well, which I’ll touch on my next post.

The RavenDB indexing process: Optimization

Wed, 18 Apr 2012 09:00:00 GMT

The actual process done by RavenDB to index documents is a fairly complex one. In order to understand what exactly happened, I decided to break it apart to pseudo code.

It looks something like this:

while database_is_running:
  stale = find_stale_indexes()
  lastIndexedEtag = find_last_indexed_etag(stale)
  docs_to_index = get_documents_since(lastIndexedEtag, batch_size)
  
  filtered_docs = execute_read_filters(docs_to_index)
  
  indexing_work = []
  
  for index in stale:
    
    index_docs = select_matching_docs(index, filtered_docs)
    
    if index_docs.empty:
      set_indexed(index, lastIndexedEtag)
    else
      indexing_work.add(index, index_docs)
      
  for work in indexing_work:
  
     work.index(work.index_docs)

And now let me show you the areas in which we did some perf work:

while database_is_running:
  stale = find_stale_indexes()
  lastIndexedEtag = find_last_indexed_etag(stale)
  docs_to_index = get_documents_since(lastIndexedEtag, batch_size)
  
  filtered_docs = execute_read_filters(docs_to_index)
  
  indexing_work = []
  
  for index in stale:
    
    index_docs = select_matching_docs(index, filtered_docs)
    
    if index_docs.empty:
      set_indexed(index, lastIndexedEtag)
    else
      indexing_work.add(index, index_docs)
      
  for work in indexing_work:
  
     work.index(work.index_docs)

All of which gives us a major boost in the system performance. I’ll discuss each part of that work in detail, don’t worry

RavenDB & FreeDB: An optimization story

Tue, 17 Apr 2012 09:00:00 GMT

So, as I noted in a previous post, we loaded RavenDB with all of the music CDs in existence (or nearly so). A total of 3.1 million disks and 43 million tracks. And we had some performance problems. But we got over them, and I am proud to give you the results:

	Old	New
Importing Data	Couple of hours	42 minutes
Raven/DocumentsByEntityName	And hour and a half	23.5 minutes
Simple index over disks	Two hours and twenty minutes	24.1 minutes
Full text index over disks and tracks	More than seven hours	37.5 minutes

Tests were run on the same machine, and the database HD was a single 300 GB 7200 RPM drive.

I then decided to take this one step further, and check what would happen when we already had the indexes. So we created three indexes. One Raven/DocumentsByEntityName, one for doing simple querying over disks and one for full text searches on top of all disks and tracks.

With 3.1 million documents streaming in, and three indexes (at least one of them decidedly non trivial), the import process took an hour and five minutes. Even more impressive, the indexing process was fast enough to keep up with the incoming data so we only had about 1.5 seconds latency between inserting a document and having it indexed. (Note that we usually seem much lower times for indexing latencies, usually in the low tens of milliseconds, when we aren’t being bombarded with documents).

Next up, and something that we did not optimize, was figuring out how costly it would be to query this. I decided to go for the big guns, and tested querying the full text search index.

Testing “Query:Adele” returned a result (from a cold booted database) in less than 0.8 seconds. But remember, this is after a cold boot. So let us see what happen when we issue a few other queries?

Query:Pearl - 0.65 seconds
Query:Abba – 0.67 seconds
Query:Queen – 0.56 seconds
Query:Smith – 0.55 seconds
Query:James – 0.77 seconds

Note that I am querying radically different values, so I force different parts of the index to load.

Querying for “Query:Adele” again? 32 milliseconds.

Let us see a few more:

Query:Adams – 0.55 seconds
Query:Abrahams – 0.6 seconds
Query:Queen – 85 milliseconds
Query:James – 0.1 seconds

Now here are a few things that you might want to consider:

We have done no warm up to the database, just started it up from cold boot and started querying.
I actually think that we can do better than this, and this is likely to be the next place we are going to focus our optimization efforts.
We are doing a query here over 3.1 million documents, using full text search.
There is no caching involved in the speed increases.

More goodies are coming in.

RavenDB & FreeDB: An optimization opportunity

Mon, 16 Apr 2012 09:00:00 GMT

Update: The numbers in this post are not relevant. I include them here solely so you would have a frame of reference. We have done a lot of optimization work, and the numbers are orders of magnitude faster now. See the next post for details.

The purpose of this post is to setup a scenario, see how RavenDB do with it, and then optimize the parts that we don’t like. This post is scheduled to go about two months after it was written, so anything that you see here is likely already fixed. In future posts, I’ll talk about the optimizations, what we did, and what was the result.

System note: I run those tests on a year old desktop, with all the database activity happening on a single 7200 RPM 300GB disk with 8 GB of RAM. Please don’t get to hung up on the actual numbers, I include them for reference, but real hardware on production system should kick this drastically higher. Another thing to remember is that this was an active system, while all of those operations were running, I was actively working and developing on the machine. The main point is to give us some sort of a metric about where we are, and to see whatever we like this or not.

We keep looking at additional things that we can do with RavenDB, and having large amount of information to tests things with is awesome. Having non fake data is even awesomer, because fake data is predictable data, while real data tend to be much more… interesting.

That is why I decided to load the entire freedb database into RavenDB and see what is happening.

What is freedb?

freedb is a database to look up CD information using the internet. This is done by a client (a freedb aware application) which calculates a (nearly) unique disc ID for a CD in your CD-Rom and then queries the database. As a result, the client displays the artist, CD-title, tracklist and some additional info.

The nice thing about freedb is that you can download their data* and make use of it yourself.

* The not so nice thing is that the data is in free form text format. I wrote a parser for it if you really want to use it, which you can find here: https://github.com/ayende/XmcdParser

So I decided to push all of this data into RavenDB. The import process took a couple of hours (didn’t actually measure, so I am not sure exactly how much), and we ended up with a RavenDB database with: 3,133,903 documents. Memory usage during the import process was ~100 MB – 150 MB (no indexes were present).

The actual size in RavenDB is 3.59 GB with 3.69 GB reserved on the file system.

Starting the database from cold boot takes about 4 seconds.

This is what the document looks like:

A full backup of the database took about 3 minutes, with all of the time dedicate for pure I/O.

Doing an export, using smuggler (on the local machine, 128 document batches) took about 18 minutes and resulted in a 803MB file (not surprising, smuggler output is a compressed file).

Note that we created this in a completely empty database, so the next step was to actually create an index and see how the database behaves. We create the default Raven/DocumentsByEntityName index, and got 5,870 seconds, so just over an hour and a half. For what it worth, this resulted in on disk index with a size of 125MB.

I then tried a much more complex index:

Just to give you some idea, this index gives you full text search support over just about every music cd that was ever made. To be frank, this index scares me, because it means that we have to have index entry for every single track in the world.

After indexing was completed, we ended up with a 700 MB on disk presence. Indexing took about 7 hours to complete. That is a lot, but remember what we are dealing with, we indexed 3.1 million documents, but we actually indexed, 52,561,894 values (remember, we index each and every track). The interesting bit is that while it took a lot of CPU (full text indexing usually does) memory usage was relatively low, it peaked about 300 MB and usually was around the 180MB).

Searching over this index is not as fast as I would like, taking about a second to complete. Then again, the results are quite impressive:

Well, given that this is the equivalent of a 52 million records (in this case, literally records ) , and we are performing full text search, quite nice.

Let us see what happens when do something a little simpler, shall we?

In this case, we are only indexing 3.1 millions documents, and we don’t do full text searches. This index took 2.3 hours to run.

Queries on that are a much more satisfactory rate of starting out at 75 ms and dropping to 5 ms very quickly.

re: Kiip’s MongoDB’s experience

Sun, 15 Apr 2012 04:29:00 GMT

We got asked several times to respond to this post, about the reason Kiip moved away from MongoDB:

On the surface, RavenDB and MongoDB are really similar, looking at the Good parts of the Kiip post, we have schemalessness, easy replication, rich query langauge and we can be access from multiple languages.

But under the hood, RavenDB operates in a completely different way than MongoDB does. A vast majority of the issues that Kiip run into are actually low level (really low level, is some cases) issues that shouldn’t really be visible to the user.

Non-counting B-Trees

The fact that MongoDB uses non counting B-Trees? The only reason that the user care about that is that it actually impacts performance, but the Kiip blog mentions a bunch of other issues related to that.

In RavenDB, we use Lucene as the indexing format, and we really don’t care about the actual format of the indexes. We natively support Count() and limit / skip, because we feel that those are actually core parts of what most users need. In fact, our API allows us to get the total count of results of a paged query as a by product of actually making the query. There isn’t any additional cost for doing this.

Poor Memory Management

MongoDB relies on the OS to do the memory management, by letting the OS memory manager to do its work. That is actually quite a smart decision, because I can guarantee that more work has gone into optimizing the OS memory manager than could have been invested by the MongoDB project. But that is just part of the work.

In RavenDB, we are actually a managed application, so we don’t have directly control over memory. That doesn’t mean that we don’t actually manage it. We have several layers of caching in place, exactly because we know more than the OS about our own usage scenarios. In many cases, even if you are making a totally new request, it would never hit the disk, because we are keeping track on hot data and making sure that it resides in memory. This applies to both indexes and documents, mind. And during the indexing process we are very careful about memory management.

Sure, the OS memory manager is more optimized, but the database knows what is going on, and can predict its own usage patterns. That is how RavenDB does a lot of magic relating to auto configuration.

Uncompressed field names

In MongoDB, it is considered good practice to shorten field names for space optimization. But MongoDB doesn’t do it for you automatically.

RavenDB doesn’t compress field names, but at the same time, it isn’t a good practice to do so. In fact, I think that this is a horrible little mess. There are a lot of arguments against compressing field names, not the least of which is that it makes it pretty hard to figure out what it is that you are actually trying to do. Looking at the raw data, something that is done fairly frequently when debugging and troubleshooting becomes harder to work with and manage:

{
  "a2": "nathan ",
  "d3": "",
  "a2": "2012-05-17T00:00:00.0000000",
  "h3": "2012-04-15T00:00:00.0000000",
  "r2": "archanid@sample.com",
  "o2": "8169cd4a-babf-4015-a3c7-4d503642e021",
  "o1": "products/NHProf"
}

Anyone wants to figure out what this document is about? And at least in this one, the data itself tells you a lot about the actual content.

There are far better alternatives in place. In RavenDB, we do full response / request compression, and we allow to do document compression on disk as well. If we were ever to get to the point where this would be a serious problem (and so far, it isn’t, even on large data sets), it would be less than a week of work to implement string interning inside RavenDB, so we would use the same string references for field values.

Global write lock

MongoDB (as of the current version at the time of writing: 2.0), has a process-wide write lock. … At this point, all other operations including reads are blocked because of the write lock.

Now, to be fair, also have a write lock, but it isn’t nearly as bad as it is in MongoDB. RavenDB write lock is actually for… writes, and it doesn’t interfere with the either reads or indexes. It is on the list of things to remove, but the crazy part is. So far, and we have really demanding users, no one cares. The reason that no one cares is that this is really small lock, and it only affects writes, it is not Stop the World type of thing.

Safe off by default

I am just going to let Kiip’s words stand for themselves (emphasis mine):

This is a crazy default, although useful for benchmarks. As a general analogy: it’s like a car manufacturer shipping a car with air bags off, then shrugging and saying “you could’ve turned it on” when something goes wrong.

RavenDB entire philosophy is around Safe by Default. That is the only thing that really make sense, because otherwise… Well… here is what happenned at Kiip:

We lost a sizable amount of data at Kiip for some time before realizing what was happening and using safe saves where they made sense (user accounts, billing, etc.).

Offline table compaction

Every now and then, you need to take down MongoDB and let it compact its on disk data. This is another Stop the World operation, and the only way to keep up when you do so is to have a hot standby ready.

RavenDB does all maintenance task while the server is up and serving requests. You don’t need any downtime just because RavenDB need to arrange some data on disk, we take care of that live, and with no interruption in service.

Secondaries do not keep hot data in RAM

As Kiip explains it:

The primary doesn’t relay queries to secondary servers, preventing secondaries from maintaining hot data in memory. This severely hinders the “hot-standby” feature of replica sets, since the moment the primary fails and switches to a secondary, all the hot data must be once again faulted into memory.

RavenDB doesn’t do so either, but for a drastically different reason. As I mentioned earlier, the way RavenDB works is quite different. When you are running a hot standby node, it will get the new data from the server and index it. We keep the index open, so for a lot of the data, it is already going to be in memory. For the rest, as I mentioned, we have several layers of caches that would help prevent needing to page gigabytes on data into memory.

Conclusion

As an utterly unbiased observer (), I can say that RavenDB rocks.

What we are actually seeing here is that RavenDB put different emphasis on different things. I really care for making the common application level scenarios easy and nice to work with. And I had enough time supporting production level apps that I tried very hard to make sure that RavenDB can take care of itself for most scenarios without any hand holding.

RavenDB Course–Israel

Sun, 01 Apr 2012 10:52:00 GMT

I got repeated calls for doing a RavenDB in Israel, and it really makes little sense not to do one here, since it is the one we would have the easiest time running.

Therefor, I am pleased to announce that our two days RavenDB course is going to open in Israel on the 11 – 12 July. We are going to do the course in our offices in Hadera, and part of the course will include interaction with the actual development team.

You can register for the course using the following link. We provide early bird registration until the 16th May.

The course is going to be in English, and is open for people from outside of Israel as well.

What have we been up to? And some future plans

Fri, 30 Mar 2012 10:00:00 GMT

We have been head down for a while, doing some really cool things with RavenDB (sharding, read striping, query intersection, indexing reliability and more). But that meant that for a while,things that are not about writing code for RavenDB has been more or less on auto-pilot.

So here are some things that we are planning. We will increase the pace of RavenDB courses and conference presentation. You can track it all in the RavenDB events page.

Conferences

Progressive NoSQL – RavenDB – 2nd generation document database.– May 9 London – I’ll be talking about RavenDB to a crowd of NoSQL enthusiastic, should be interesting, especially the compare & contrast part.
Sharding and Scaling with RavenDB – May 15, London – Prepare for some truly awesome demos.
DevSum 20120 - RavenDB and the Amazing Features – May 23 , Stockholm, I am going to try and go through some of the more cool features of RavenDB, stuff that is going to make you go gaga wanting to use it.
NDC 2012 – RavenDB Data Modeling – June 7, Oslo. Itamar is going to go over how to model data in RavenDB. This is one of the greatest challenges people moving over to RavenDB from RDBMS are facing, and Itamar is going to cover common scenarios and walkthrough about how to approach the problem.
Ordev 2012 - Sharding and Scaling with RavenDB – Nov 5, Malmo.

RavenDB Courses

2 Days RavenDB Workshop – Oren Eini – May 14, London.
2 Days RavenDB Workshop – Oren Eini – May 21, Stockholm.
2 Days RavenDB Workshop – Itamar Syn-Hershko – June 4, Oslo.
2 Days RavenDB Workshop – Oren Eini – June 28, Malmo.
2 Days RavenDB Workshop – Itamar Syn-Hershko – Aug 21, New York.
2 Days RavenDB Workshop – Itamar Syn-Hershko – Sep 12, London.

NHibernate Courses

3 Days NHibernate 3.x Workshop – Oren Eini, May 16, London.

Not finalized yet

August 2012.
- User groups talks in Philadelphia & Washington DC by Itamar Syn-Hershko.
- One day boot camp for moving from being familiar with RavenDB to being a master in Chicago.
September 2012.
- RavenDB Course in Austin, Texas.

Consulting Opportunities

We are also available for on site consulting in the following locations and times. Please contact us directly if you would like to arrange for one of RavenDB core team to show up at your door step. Or if you want me to do architecture or NHibernate consulting.

Oren Eini – Malmo, June 26 – 27.
Oren Eini – Berlin, July 2 – 4.
Itamar Syn-Hershko – New York, Aug 23.
Itamar Syn-Hershko – Chicago, Aug 30 or Sep 3.
Itamar Syn-Hershko – Austin, Sep 3.
Itamar Syn-Hershko – Toronto, Sep 9 – 10.
Itamar Syn-Hershko – London, Sep 11.