This is just some logistical cleanups.

The code for the entire series can be found here: https://github.com/ayende/nuget.perf

No, I’ll not do a similar SQL version, if you want to, I would be very interested in seeing one, but that isn’t something that I intend to do.

Yes, it is a simple and trivial implementation, but that was pretty much the whole point. Being able to get to that scale without actually doing anything special is what we strive for in RavenDB.

After seeing how well RavenDB does in perf testing, I decided to take it up a notch.

• Starting from 10 users, with a  step duration of 1 sec, add 50 users for each step, all the way to 3,000.
• Start with a warm up period of 20 seconds, then run the test for 10 minutes.

Let us see what happens, okay?

Just to be clear, this is a RavenDB application running with three thousands concurrent users, on an off the shelve laptop while I was busy doing other stuff.

One word of warning before hand, because I run everything on a single machine, just running so many users on the machine significantly slowed down how RavenDB is reacting. Basically, the code for managing the perf test took so many resources that RavenDB had to fight to get some to actually answer the queries.

Scared yet, because here are the results in graph form.

Now you can actually see that we have some fluctuations in the graphs, the number of users grows and grows until we get to 3,000 and we have 0.37 seconds response times.

Again, I remind you, we have done zero optimizations and this is idiomatic RavenDB code. And we were able to serve requests at a frankly pretty amazing rate of speed.

And here are they in their full details:

Note that the reason fro the high CPU usage is that the tests and RavenDB were running on the same machine.

The test was run locally (no network involved ) on a Lenovo W520 laptop with 8 cores & 8 GB RAM with an SSD card. The storage engine we used was Esent, Safe Transactions. Default RavenDB configuration, running in console, with logging disabled.

We took the most obvious approach both in the code we wrote and the test approach. I am pretty sure that I’ll get a lot of helpful suggestions about the load testing. The code is available here, and you are more than welcome to take it for a spin and get your own results. What is important for me to note is that we have done exactly zero performance tuning. That is relevant to both the index we use, to the code that we wrote, everything. I just wrote things down, and didn’t worry about performance, even though this code is going to go through a load test.

Why don’t I worry about it? Because RavenDB is setup to do the Right Thing. It will self optimize itself without you need to take care of that.

With that said, here are the test results:

You can see that the red line is the number of users we have, and we have this worrying green line that seems to go crazy…

Except that this is actually the number of page served. The part that we care about is actually the Avg. Page Time, and that is the blue line.

This line, however, is basically flat no matter the load.

Here are the test results in details

You can dig in and look at the data, it is quite interesting. Under the load of 300 users, the average page response time was… 0.0062 seconds.

And RavenDB was using just 13% of the CPU, and that include running the agents running the tests.

In my next post, we will go totally crazy…

For the tests, we used VS 2012 load testing tool.

We defined the following tests:

Just browsing through the packages listing:

Browsing a bit then searching, and then narrowing the search:

And finally, searching a few packages by their id, tags, etc:

I then defined the following load test:

With the following distribution:

Finally, we have the way we actually run the test:

We get 20 seconds of warm up, then 5 minutes of tough load.

On my next post, we will see how we did.

So, after talking so long about the perf issues, here is the final part of this series. In which we actually take this for a spin using Load Testing.

I built a Web API application to serve as the test bed. It has a RavenController, which looks like this:

public class RavenController : ApiController
{
    private static IDocumentStore documentStore;

    public static IDocumentStore DocumentStore
    {
        get
        {
            if (documentStore == null)
            {
                lock (typeof (RavenController))
                {
                    if (documentStore != null)
                        return documentStore;
                    documentStore = new DocumentStore
                        {
                            Url = "http://localhost:8080",
                            DefaultDatabase = "Nuget"
                        }.Initialize();
                    IndexCreation.CreateIndexes(typeof (Packages_Search).Assembly, documentStore);
                }
            }
            return documentStore;
        }
    }

    public IDocumentSession DocumentSession { get; set; }

    public override async Task<HttpResponseMessage> ExecuteAsync(HttpControllerContext controllerContext, CancellationToken cancellationToken)
    {
        using (DocumentSession = DocumentStore.OpenSession())
        {
            HttpResponseMessage result = await base.ExecuteAsync(controllerContext, cancellationToken);
            DocumentSession.SaveChanges();
            return result;
        }
    }
}

And now we have the following controllers:

public class PackagesController : RavenController
{
    public IEnumerable<Packages_Search.ReduceResult> Get(int page = 0)
    {
        return DocumentSession.Query<Packages_Search.ReduceResult, Packages_Search>()
            .Where(x=>x.IsPrerelease == false)
            .OrderByDescending(x=>x.DownloadCount)
                .ThenBy(x=>x.Created)
            .Skip(page*30)
            .Take(30)
            .ToList();
    }
}

public class SearchController : RavenController
{
    public IEnumerable<Packages_Search.ReduceResult> Get(string q, int page = 0)
    {
        return DocumentSession.Query<Packages_Search.ReduceResult, Packages_Search>()
            .Search(x => x.Query, q)
            .Where(x => x.IsPrerelease == false)
            .OrderByDescending(x => x.DownloadCount)
                .ThenBy(x => x.Created)
            .Skip(page * 30)
            .Take(30)
            .ToList();
    }
}

And, just for completeness sake, the Packages_Search index looks like this:

public class Packages_Search : AbstractIndexCreationTask<Package, Packages_Search.ReduceResult>
{
    public class ReduceResult
    {
        public DateTime Created { get; set; }
        public int DownloadCount { get; set; }
        public string PackageId { get; set; }
        public bool IsPrerelease { get; set; }
        public object[] Query { get; set; }
    }

    public Packages_Search()
    {
        Map = packages => from p in packages
                          select new
                              {
                                  p.Created, 
                                  DownloadCount = p.VersionDownloadCount, 
                                  p.PackageId, 
                                  p.IsPrerelease,
                                  Query = new object[] { p.Tags, p.Title, p.PackageId}
                              };
        Reduce = results =>
                 from result in results
                 group result by new {result.PackageId, result.IsPrerelease}
                 into g
                 select new
                         {
                             g.Key.PackageId,
                             g.Key.IsPrerelease,
                             DownloadCount = g.Sum(x => x.DownloadCount),
                             Created = g.Select(x => x.Created).OrderBy(x => x).First(),
                             Query = g.SelectMany(x=>x.Query).Distinct()
                         };

        Store(x=>x.Query, FieldStorage.No);
    }
}

That is enough setup, in the next post, I’ll discuss the actual structure of the load tests.

I hope this is the last one, because I can never recall what is the next Latin number.

At any rate, it has been pointed out to me that I made an error in importing the data. I assumed that the DownloadCount field that I got from the Nuget API is the download count for the specific package, but it appears that this is the total downloads count, across all versions of this package. The actual download number for a specific package is: VersionDownloadCount.

That changes things a bit, because the way Nuget sorts things is based on the total download count, not the download count for a specific version. The reason this complicate things is that we aren’t going to store the total download count in all the version documents. First, let us see the sort of query we need to write. In SQL, it would look like this:

select top 30 skip 30 
    Id,
    PackageId,
     Created, 
    (select sum(VersionDownloadCount) from Packages all where all.PackageId = p.PackageId) as TotalDownloadsCount
from Packages p
where IsPrerelease = 0
order by TotalDownloadsCount desc, Created

This is a much simplified version of the real query, and something that you can’t actually write this simply in SQL, most probably. But it gets the point.

Note that in order to process this query, the RDMBS would have to first aggregate all of the data (for each row, mind) then do the paging, then give you the results. Sure, you can keep a counter for all the downloads for a package, but considering the fact that downloads are highly parallel and happen all the time, waiting for writers to finish doing their update.

Instead, with RavenDB, we are going to use a map/reduce index and query on that.

This should be fairly simple to follow. In the map we go over all the packages, and output their package id, whatever they have been released, the specific version download count and the date it was created.

In the reduce, we group by the package id and whatever is was pre released or not ( I am assuming that we usually don’t want to show the pre-release stuff there).

Finally, we sum up all of the individual package downloads and we output the oldest created date. Using all of that, we can now move to the next step, and actually query that:

There  is a small bug here, since I don’t see RavenDB in the results,  but I guess I’ll have to wait until I get the updated data from Nuget.

Actually, that is not quite true, for pre-released software, we are pretty high up:

That explains much, RavenDB 1.2 is pretty awesome.

So far, we have been focusing on various ways to get the raw results from RavenDB. What are the packages that match your queries, and whatever we can be really smart about it.

But let us say that we got the results that we wanted, this is still just half the work, because we can give the user additional information about those results. In particular, in this post I am going to talk about facets.

Facets are a way to provide easily understood context to a search, allowing the user to narrow down what he is looking for quickly. In our case, let us take a look what it takes to add facet supports to our NuGet console app. The first thing to do, of course, is to actually define the facets we want to work on. In this case, we care only for the Tags:

using (var session = store.OpenSession())
{
    session.Store(new FacetSetup
        {
            Id = "facets/PackagesTags",
            Facets =
            {
                new Facet
                    {
                        Name = "Tags",
                        MaxResults = 4,
                        Mode = FacetMode.Default,
                        TermSortMode = FacetTermSortMode.HitsDesc
                    }
            },
        });
    session.SaveChanges();
}

When doing facet search using this document, we will use the Tags field, using a value per each term found. We want to get the top 4, sorted by their hits.

And here is how we are actually doing the faceted query:

var facetResults = q.ToFacets("facets/PackagesTags");
foreach (var result in facetResults.Results)
{
    Console.WriteLine();
    Console.Write("{0}:\t", result.Key);
    foreach (var val in result.Value.Values)
    {
        Console.Write("{0} [{1:#,#}] | ", val.Range, val.Hits);
    }
    Console.WriteLine();
}

It is a one liner, with all of the rest of the code dedicated to just printing things out.

Finally, here are the results:

As you can see, searching for “dal”, we can narrow the searches for linq, orm, etc. Searching for events, we get reactive extensions, etc.

Using facets gives the user additional information about his search (including things like, am I close to what I want), discoverability over your dataset and additional tools to explore it.

All in all, I think that this is a pretty neat thing.

So far, we imported the NuGet data to RavenDB and seen how we can get it out for the packages page and then looked into how we can utilize RavenDB features to help us in package search. I think we did a good job there, but we can probably do better still. In this post, I am going to stop showing off things in the Studio and focus on code. In particular, advanced searching options.

We will start from the simplest search possible. Or not, because we are doing full text search and quite a few other things aside even in the base line search. Anyway, here is the skeleton program:

while (true)
{
    Console.Write("Search: ");
    var search = Console.ReadLine();
    if(string.IsNullOrEmpty(search))
    {
        Console.Clear();
        continue;
    }
    using (var session = store.OpenSession())
    {
        var q = session.Query<PackageSearch>("Packages/Search")
            .Search(x => x.Query, search)
            .Where(x => x.IsLatestVersion && x.IsAbsoluteLatestVersion && x.IsPrerelease == false)
            .As<Package>()
            .OrderByDescending(x => x.DownloadCount).ThenBy(x => x.Created)
            .Take(3);
        var packages = q.ToList();

        foreach (var package in packages)
        {
            Console.WriteLine("\t{0}", package.Id);
        }
    }
}

Now, we are going to run this and see what we get.

So far, so good. Now let us try to improve things. What happens when we search for “jquryt”? Nothing is found, and that is actually pretty sad, because to a human, it is obvious what you are trying to search on.

If you have fat fingers and have a tendency to creatively spell words, I am sure you can emphasize with this feeling. Luckily for us, RavenDB is going to help, let us see how:

What?!

How did it do that? Well, let us look at the changes in the code, shall we?

private static void PeformQuery(IDocumentSession session, string search, bool guessIfNoResultsFound = true)
{
    var packages = session.Query<PackageSearch>("Packages/Search")
        .Search(x => x.Query, search)
        .Where(x => x.IsLatestVersion && x.IsAbsoluteLatestVersion && x.IsPrerelease == false)
        .As<Package>()
        .OrderByDescending(x => x.DownloadCount).ThenBy(x => x.Created)
        .Take(3).ToList();

    if (packages.Count > 0)
    {
        foreach (var package in packages)
        {
            Console.WriteLine("\t{0}", package.Id);
        }
    }
    else if(guessIfNoResultsFound)
    {
        DidYouMean(session, search);
    }
    else
    {
        Console.WriteLine("\tNo search results were found");
    }
}

The only major change was the call to DidYouMean(), so let us see what is going on in there.

private static void DidYouMean(IDocumentSession session, string search)
{
    var suggestionQueryResult = session.Query<PackageSearch>("Packages/Search")
        .Search(x => x.Query, search)
        .Suggest();
    switch (suggestionQueryResult.Suggestions.Length)
    {
        case 0:
            Console.WriteLine("\tNo search results were found");
            break;
        case 1:
            // we may have it filtered because of the other conditions, don't recurse again

            Console.WriteLine("\tSearch corrected to: {0}", suggestionQueryResult.Suggestions[0]);
            Console.WriteLine();

            PeformQuery(session, suggestionQueryResult.Suggestions[0], guessIfNoResultsFound: false);
            break;
        default:
            Console.WriteLine("\tDid you mean?");
            foreach (var suggestion in suggestionQueryResult.Suggestions)
            {
                Console.WriteLine("\t - {0} ?", suggestion);
            }
            break;
    }
}

Here, we ask RavenDB, “we couldn’t find anything what we had, can you give me some other ideas?” RavenDB can check the actual data that we have on disk and suggest similar alternative.

In essence, we asked RavenDB for what is nearby, and it provided us with some useful suggestions. Because the suggestions are actually based on the data we have in the db, searches on that will produce correct results.

Note that we have three code paths here, if there is one suggestion, we are going to select that immediately. Let us see how this looks like in practice:

Users tend to fall in love with those sort of features, and with RavenDB you can provide them in just a few lines of code and absolutely no hassle.

In my next post (and probably the last in this series) we will discuss even more awesome search features .

Now we get to the good parts, actually doing searches for Packages, not just showing them in packages page, but doing complex and interesting searches. The current (after optimization) query looks like this:

SELECT        TOP (30)
       -- fields removed for brevity
FROM        (

            SELECT        Filtered.Id
                    ,    Filtered.PackageRegistrationKey
                    ,    Filtered.Version
                    ,    Filtered.DownloadCount
                    ,    row_number() OVER (ORDER BY Filtered.DownloadCount DESC, Filtered.Id ASC) AS [row_number]
            FROM        (
                        SELECT        PackageRegistrations.Id
                                ,    Packages.PackageRegistrationKey
                                ,    Packages.Version
                                ,    PackageRegistrations.DownloadCount
                        FROM        Packages
                        INNER JOIN    PackageRegistrations ON PackageRegistrations.[Key] = Packages.PackageRegistrationKey
                        WHERE        ((((Packages.IsPrerelease <> cast(1 as bit)))))
                                ((((AND    Packages.IsLatestStable = 1))))
                                ((((AND    Packages.IsLatest = 1))))
                                AND    (
                                        PackageRegistrations.Id LIKE '%jquery%' ESCAPE N'~'
                                    OR    PackageRegistrations.Id LIKE '%ui%' ESCAPE N'~'

                                    OR    Packages.Title LIKE '%jquery%' ESCAPE N'~'
                                    OR    Packages.Title LIKE '%ui%' ESCAPE N'~'

                                    OR    Packages.Tags LIKE '%jquery%' ESCAPE N'~'
                                    OR    Packages.Tags LIKE '%ui%' ESCAPE N'~'
                                    )
                        ) Filtered
            ) Paged
INNER JOIN    PackageRegistrations ON PackageRegistrations.[Key] = Paged.PackageRegistrationKey
INNER JOIN    Packages ON Packages.PackageRegistrationKey = Paged.PackageRegistrationKey AND Packages.Version = Paged.Version
WHERE        Paged.[row_number] > 30
ORDER BY    PackageRegistrations.DownloadCount DESC
        ,    Paged.Id

I can hear the DB whimpering in fear in a dark corner, where it is hiding while it isn’t being flogged by cruel and unusual queries.

Okay, there is  a certain amount of hyperbole here, I’ll admit .But at least it is funny.

At any rate, here we have query that allows the user to search for the latest stable packages by their id, title or tags. To make things interesting for the DB, all queries are using ‘%jquery%’ form. This is something that particularly every single resource you can find about databases will warn you against. You can read why here. I think we can safely assume that the NuGet guys do not use EF Prof, or they wouldn’t go this route.

Actually, I am being unfair here. There really aren’t many other good options when you start to need those sort of things. Yes, I know of SQL Server Full Text Indexes, they are complex to setup and maintain and they don’t provide enough facilities to do interesting stuff. They are also more complex to program against. You could maintain your own indexes on the side (Lucene, Fast, etc). Now you have triple the amount of work that you have to do, and care and maintenance of those isn’t trivial. For either the devs or the ops team.

So I can certainly follow why the decision was make to use LIKE ‘%jquery%’, even though it is a well known problem.

That said, it is the wrong tool for the job, and I think that RavenDB can do a lot more and in more interesting ways as well.

Let us see the index that can handle these sort of queries.

What does this index do?

Well, it index the a bunch of fields to allow them to be searched for by value, but it also do something else that is query interesting. The Query field in the index takes information from several different fields that are all indexed as one. We also specify that this index will treat the Query field as the target for full text analysis. This means that we can now write the following query:

In code, this would look like this:

var results = session.Query<Package_Search.Request, Package_Search>()
    .Where(x=> x.IsLatestVersion && x.IsAbsoluteLatestVersion && x.IsPrerelease == false)
    .Search(x=>x.Query, userSearchTerms)
    .OrderByDescending(x=>x.DownloadCount).ThenBy(x=>x.Created)
    .Take(30
    .As<Package>()
    .ToList();

This will generate the query you can see above, and return the first 30 results.

But a lot more is actually happening here, let us look at what actually goes on in the index:

Here you can see the actual terms that were indexed in the database for each of the documents. The reason that this is important is that when it comes the time to do searches, we aren’t going to need to do anything as crass as a full table scan, which is what SQL has to do. Instead, all of those terms are located in an index, and we have the <<jquery ui>> search string. We can them do a very simple index lookup (cost of that is O(logN), if you’ll recall) to find your results.

And of course, we have this guy:

So I am pretty happy about this so far, but we can probably do better. We will see how in our next post.

Before we move on to discussing how to implement package search, I wanted to take a bit of time to discuss the we structured the data. In particular, there are a bunch of properties that feel very relational in nature. In particular, these two properties:

• Tags: Ian_Mercer Natural_Language Abodit NLP
• Dependencies: AboditUnits:1.0.4|Autofac.Mef:2.5.2.830|ImpromptuInterface:5.6.2|log4net:1.2.11

In the current version of NuGet, those properties are actually stored as symbol separated strings. The reason for that? In relational databases, if you want to have a collection, you have to have another table, then join to it, then take care of it, and wake up in the middle of the night to take it to a walk. So people go the obvious route and just concatenate strings and hope for the best. Note that in the dependencies case, we have multi level concatenation.

In RavenDB, we have full fledged support for storing complex objects, so the tags above will become:

And what about the dependencies? Those we store in an array of complex objects, like so:

RavenDB allows us to store the model in a way that is easy on the eye ,natural to work with and in general making our lives easier.

Let us say that I wanted to add a feature to NuGet, “show me all the packages that use this package”?

And allow me to brag a little bit?

By the way, just to be sure that everyone has full grasp about what is going on, I am writing this post while on 30,000 feet. The laptop I am using is NOT connected to power, and the data set that I am using is the full NuGet dataset.

Compare the results you get from RavenDB to what you have to do in SQL: Dependencies LIKE ‘%log4net%’

You can kiss your performance goodbye with these sort of queries.