Ayende @ Rahien

Refunds available at head office

Inside RavenDB 3.0

I’ve been working for a while on seeing where we can improve RavenDB, and one of the things that I wanted to address is having an authoritative source to teach people about RavenDB. Not just documentation, those are very good for reference, but not so good to give you a guided tour and actually impart knowledge. That is what I wanted to do, to take the last five years or so of working on and with RavenDB and distill them.

The result is about a hundred pages or so (and likely to be three or four hundred pages). In other words, I slipped up and started churning out a book Smile.

You can download the alpha version using the following link (which will be valid for the next two weeks). I want to emphasis that this is absolutely unedited, and there are likely to be error for zpelling in grammar*. Those will be fixed down the line, currently I’m mostly focused on getting the content out. Here is also the temporary cover.

Cover

Comments are welcome. And yes, this will be an actual book, in the end, which you can hold in your hand and hopefully beat someone over the head if they need to smarten up.

* The errors in that particular sentence were intentional.

Tags:

Published at

Originally posted at

Comments (8)

NHibernate 4.0 released!

You can get it here!

This has a lot of bug fixes, fitting into the .NET 4.x eco system and in general good stuff.

Probably my favorite one is the SQL Server 2012 paging, which is so much easier to understand.

Happy hibernating…

Tags:

Published at

Originally posted at

Comments (2)

It is a good day, celebrate it

It is a good day, so I decided to share some joy.

For today only, we offer 21% discount for all our products. You can get that using coupon code: bzeiglglay

This applies to RavenDB (Standard, Enterprise and ISV), RavenDB Professional & Production Support and NHibernate Profiler and Entity Framework Profiler

This offer will be valid for 24 hours only.

Published at

Originally posted at

Comments (4)

Question 6 is a trap, a very useful one

In my interview questions, I give candidates a list of 6 questions. They can either solve 3 questions from 1 to 5, or they can solve question 6.

Stop for a moment and ponder that. What do you assume that relative complexity of those questions?

 

 

 

 

 

 

Questions 1 –5 should take anything between 10 – 15  minutes to an hour & a half, max. Question 6 took me about 8 hours to do, although that included some blogging time about it.

Question 6 require that you’ll create an index for a 15 TB CSV file, and allow efficient searching on it.

While questions 1 – 5 are basically gate keeper questions. If you answer them correctly, we’ve a high view of you and you get an interview, answering question 6 correctly pretty much say that we past the “do we want you?” and into the “how do we get you?”.

But people don’t answer question 6 correctly. In fact, by this time, if you answer question 6, you have pretty much ruled yourself out, because you are going to show that you don’t understand something pretty fundamental.

Here are a couple of examples from the current crop of candidates. Remember, we are talking about a 15 TB CSV file here, containing about 400 billion records.

Candidate 1’s code looked something like this:

foreach(var line in File.EnumerateAllLines("big_data.csv"))
{
       var fields = line.Split(',');
       var email = line[2]
       File.WriteAllText(Md5(email), line);
}

Plus side, this doesn’t load the entire data set to memory, and you can sort of do quick searches. Of course, this does generate 400 billion files, and takes more than 100% as much space as the original file. Also, on NTFS, you have a max of 4 billion files per volume, and other FS has similar limitations.

So that isn’t going to work, but at least he had some idea about what is going on.

Candidate 2’s code, however, was:

// prepare
string[] allData = File.ReadAllLines("big_data.csv");
var users = new List<User>();
foreach(var line in allData)
{
     users.Add(User.Parse(line));
}
new XmlSerializer().Serialize(users, "big_data.xml");

// search by:

var users = new XmlSerialize().Deserialize("big_data.xml") as List<User>()
users.AsParallel().Where(x=>x.Email == "the@email.wtf");

So take the 15 TB file, load it all to memory (fail #1), convert all 400 billion records to entity instances (fail #2), write it back as xml (fail #3,#4,#5). Read the entire (greater than) 15 TB XML file to memory (fail #6), try to do a parallel brute force search on a dataset of 400 billion records (fail #7 – #400,000,000,000).

So, dear candidates 1 & 2, thank you for making it easy to say, thanks, but no thanks.

Tags:

Published at

Originally posted at

Comments (21)

Troubleshooting, when F5 debugging can’t help you

You might have noticed that we have been doing a lot of work on the operational side of things. To make sure that we give you as good a story as possible with regards to the care & feeding of RavenDB. This post isn’t about this. This post is about your applications and systems, and how you are going to react when !@)(*#!@(* happens.

In particular, the question is what do you do when this happens?

This situation can crop up in many disguises. For example, you might be seeing a high memory usage in production, or experiencing growing CPU usage over time, or see request times go up, or any of a hundred and one different production issues that make for a hell of a night (somehow, they almost always happen at nighttime)

Here is how it usually people think about it.

The first thing to do is to understand what is going on. About the hardest thing to handle in this situations is when we have an issue (high memory, high CPU, etc) and no idea why. Usually all the effort is spent just figuring out what and why.. The problem with this process for troubleshooting issues is that it is very easy to jump to conclusions and have an utterly wrong hypothesis. Then you have to go through the rest of the steps to realize it isn’t right.

So the first thing that we need to do is gather information. And this post is primarily about the various ways that you can do that. In RavenDB, we have actually spent a lot of time exposing information to the outside world, so we’ll have an easier time figuring out what is going on. But I’m going to assume that you don’t have that.

The end all tool for this kind of errors in WinDBG. This is the low level tool that gives you access to pretty much anything you can want. It is also very archaic and not very friendly at all. The good thing about it is that you can load a dump into it. A dump is a capture of the process state at a particular point in time. It gives you the ability to see the entire memory contents and all the threads. It is an essential tool, but also the last one I want to use, because it is pretty hard to do so. Dump files can be very big, multiple GB are very common. That is because they contain the full memory dump of the process. There is also mini dumps, which are easier to work with, but don’t contain the memory dump, so you can watch the threads, but not the data.

The .NET Memory Profiler is another great tool for figuring things out. It isn’t usually so good for production analysis, because it uses the Profiler API to figure things out, but it has a wonderful feature of loading dump files (ironically, it can’t handle very large dump files because of memory issuesSmile) and give you a much nicer view of what is going on there.

For high CPU situations, I like to know what is actually going on. And looking at the stack traces is a great way to do that. WinDBG can help here (take a few mini dumps a few seconds apart), but again, that isn’t so nice to use.

Stack Dump is a tool that takes a lot of the pain away for having to deal with that. Because it just output all the threads information, and we have used that successfully in the past to figure out what is going on.

For general performance stuff “requests are slow”, we need to figure out where the slowness actually is. We have had reports that run the gamut from “things are slow, client machine is loaded” to “things are slow, the network QoS settings throttle us”. I like to start by using Fiddler to start figuring those things out. In particular, the statistics window is very helpful:

image

The obvious things are the bytes sent & bytes received. We have a few cases where a customer was actually sending 100s of MB in either of both directions, and was surprised it took some time. If those values are fine, you want to look at the actual performance listing. In particular, look at things like TCP/IP connect, time from client sending the request to server starting to get it, etc.

If you found the problem is actually at the network layer, you might not be able to immediately handle it. You might need to go a level or two lower, and look at the actual TCP traffic. This is where something like Wire Shark comes into play, and it is useful to figure out if you have specific errors at  that level (for example, a bad connection that cause a lot of packet loss will impact performance, but things will still work).

Other tools that are very important include Resource Monitor, Process Explorer and Process Monitor. Those give you a lot of information about what your application is actually doing.

One you have all of that information, you can form a hypothesis and try to test it.

If you own the application in question, the best way to improve your chances of figuring out what is going on is to add logging. Lots & lots of logging. In production, having the logs to support what is going on is crucial. I usually have several levels of logging. For example, what is the traffic in/out of my system. Next there is the actual system operations, especially anything that happens in the background. Finally, there are the debug/trace endpoints that will expose internal state and allow you to tweak various things at runtime.

Having good working knowledge on how to properly utilize the above mention tools is very important, and should be considered to be much more useful than learning a new API or a language feature.

There is no WE in a Job Interview

This is a pet peeve of mine. When interviewing candidates, I’m usually asking some variant of “tell me about a feature you developed that you are proud of”. I’m using this question to gauge several metrics. Things like what is the candidate actually proud of, what was he working on, are they actually proud of what they did?

One of the more annoying tendencies is  for a candidate to give a reply in the form of “what we did was…”. In particular if s/he goes on to never mention something that s/he specifically did. And no, “led the Xyz team in…” is a really bad example.  I’m not hiring your team, in which case I might actually be interested in that. I’m actually interested in the candidate, personally. And if the candidate won’t tell me what it was that s/he did, I’m going to wonder if they played Solitaire all day.

Tags:

Published at

Originally posted at

Comments (23)

A tale of two interviews

We’ve been trying to find more people recently, and that means sifting trouble people. Once that process is done, we ask them to come to our offices for an interview. We recently had two interviews from people that were diametrically opposed to one another. Just to steal my own thunder, we decided not to go forward with either one of them. Before inviting them to an interview, I have them do a few coding questions at home. Those are things like:

  • Given a big CSV file (that fit in memory), allow to speedily query by name or email. The application will run for long period of time, and startup time isn’t very important.
  • Given a very large file (multiple TB), detect what 4MB ranges has changed in the file between consecutive runs of your program.

We’ll call the first one Joe. Joe has a lot of experience, he has been doing software for a long time, and has already had the chance to be a team lead in a couple of previous positions. He sent us some really interesting code. Usually I get a class or three in those answers. In this case, we got something that looked like this:

The main problem I had with his code is just finding where something is actually happening. I discarded the over architecture as someone who is trying to impress in an interview, “See all my beautiful ARCHITECTURE!”, and look at the actual code to actually do the task at hand, which wasn’t bad. 

Joe was full of confidence, he was articulate and well spoken, and appear to have a real passion for the architecture he sent us. “I’ve learned that it is advisable to put proper architecture first” and “That is now my default setting”. I disagree with those statements, but I can live with that.  What bothered me was that we asked a question along the way of “how would you deal with a high memory situation in an application”. What followed was several minutes of very smooth talk about motivating people, giving them the kind of support they need to do the job, etc. Basically, about the only thing it was missing was a part on “the Good of the People” and I would have considered whatever to vote for him. What was glaringly missing in my point of view was anything concrete and actionable.

On the other hand, we have Moe. He is a bit younger, but he already worked with NoSQL databases, which was a major plus. Admittedly, that was as a user of, instead of a developer of, but you can’t have it all. Moe’s code made me sit up and whistle. I setup an interview for the very next day, because looking at the code, there was someone there I wanted to talk to. It was very much to the point, and while it had idiosyncrasies, it showed a lot of promise. Here is the architecture for Moe’s code:

So Moe shows up at the office ,and we start the interview process. And right from the get go it is obvious that Moe is one of those people who don’t do too well in stressful situations like interviews. That is part of the reason why we ask candidates to write code at home. Because it drastically reduce the level of stress they have to deal with.

So I start talking, telling about the company and what we do. The idea is that hopefully this gives him time to compose himself. Then I start asking questions, and he gives mostly the right answers, but I’m lacking focus. I’m assuming that this is probably nervousness, so I bring up his code and go over that with him. He is much more comfortable talking about that. He had a O(logN) solution at one point, and I had to drive him toward an O(1) solution for the same problem, but he got there fairly quickly.

I then asked him what I considered to be a fairly typical question: “What areas you have complete mastery at?” This appear to have stumped him, since he took several minutes to give an answer which basically boiled down to “nothing”.

Okay… this guy is nervous, and he is probably under estimating himself, so let us try to focus the question. I asked whatever he was any good with HTML5 (not at all), then whatever he was good with server side work (have done some work there, but not an expert), and how he would deal with a high memory situation (look at logs, but after that he was stumped). When asked about the actual code he wrote for our test, he said that this was some of the hardest tasks he ever had to deal with.

That summed up to promising, but I’ve a policy of believing people when they tell me bad things about themselves. So this ended up being a negative, which was very frustrating.

The search continues…

Tags:

Published at

Originally posted at

Comments (14)

Complex indexing, simplified

RavenDB indexes are Turing complete, which means that you can do whatever you want with them. This is a very powerful feature, but it also come with a heavy burden. You can get yourself into some serious trouble. Take a look at this index:

image

 

We run into it during a troubleshooting session with a customer. And it was frankly quite hard to figure out what was going on.

Luckily, I could just throw this into RavenDB 3.0, and look at the indexing options:

image

This turned the above index into this:

image

Which was much clearer, but we could improve it a bit by removing the into clauses, so I ended up with:

image

Now, just from the following, can someone tell me what is the likely issue with this kind of index?

Tags:

Published at

Originally posted at

Comments (8)

We’re hiring… come work for us

unnamed

It seems that recently we have been going in rounds. Get more people, train them, and by the time they are trained, we already need more people Smile.

I guess that this is a good problem to have. At any rate, we are currently looking for an experience well rounded developer.

This job availability is for our offices in Hadera, Israel. If you aren’t from Israel, this isn’t for you.

This job is primarily for work on our Profilers line of products. Here is the laundry list:

  • Awesome .NET skills
  • Experience in UI development using WPF, MVVM style
  • Understanding how computers work and how to make them dance
  • History with concurrency & multi threading (concurrent work history not required)
  • Architecture / design abilities

I would like to see open source history, or projects that you can share (in other words, your projects, not employer’s code that you try to give to look at).

Please contact us at jobs@hibernatingrhinos.com if you are interested.

Published at

Originally posted at

Digging into MSMQ

I got into a discussion online about MSMQ and its performance. So I decided to test things out.

What I want to do is to check a few things, in particular, how much messages can I push to and from MSMQ in various configurations.

I created a non transactional queue, and then I run the following code:

var sp = Stopwatch.StartNew();
int count = 0;
while (sp.Elapsed.TotalSeconds < 10)
{
var message = new Message
{
BodyStream = new MemoryStream(data)
};
queue.Send(message);
count++;
}

Console.WriteLine(sp.Elapsed);
Console.WriteLine(count);

This gives me 181,832 messages in 10 seconds ,or 18,183 messages per second. I tried doing the same in a multi threaded fashion, with 8 threads writing to MSMQ, and got an insufficient resources error, so we’ll do this all in a single threaded tests.

Next, the exact same code, but for the Send line, which now looks like this:

queue.Send(message, MessageQueueTransactionType.Single);

This gives me 43,967 messages in 10 seconds, or 4,396 messages per second.

Next I added DTC, which gave me a total of 8,700 messages in ten seconds, or 870 messages per second! Yeah, DTC is evil.

Now, how about reading from it? I used the following code for that:

while (true)
{
try
{
Message receive = queue.Receive(TimeSpan.Zero);
receive.BodyStream.Read(data, 0, data.Length);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}
}

Reading from transactional queue, we get 5,955 messages per second for 100,000 messages. And using non transaction queue it can read about 16,000 messages a second.

Note that those are pretty piss poor “benchmarks”, they are intended more to give you a feel for the numbers than anything else.  I’ve mostly used MSMQ within the context of DTC, and it really hit the performance hard.

Tags:

Published at

Originally posted at

Comments (8)

On site Architecture & RavenDB consulting availabilities: Malmo & New York City

I’m going to have availability for on site consulting in Malmo, Sweden  (17 Sep) and in New York City, NY (end of Sep – beginning of Oct).

If you want me to come by and discuss what you are doing (architecture, nhibernate or ravendb), please drop me a line.

I’m especially interested in people who need to do “strange” things with data and data access. We are building a set of tailored database solutions for customers now, and we have seem customers show x750 improvement in performance when we gave them a database that was designed to fit their exact needs, instead of having to contort their application and their database to a seven dimensional loop just to try to store and read what they needed.

New RavenDB site design poll

As part of the 3.0 release, we are also going to do a full redesign of our website, and we would like to have your opinion on the matter.

Please take a look at the options and vote for your favorite: http://99designs.com/web-design/vote-u95dcs

Note, we’ll be changing the studio look & feel to match the website as well.

Tags:

Published at

Originally posted at

RavenDB 3.0 Ops: Live Tracing & Logging for production

You might have noticed a theme here Smile in where we are pushing RavenDB 3.0. This is actually an effect of how we structured our work plans, we did a lot of the new features (Voron, for example) early on, because they require a lot of time to mature. We now mostly complete the work related to user interface and exposing operational data.

This feature comes back to the old black box issue. What is the system doing? Usually, you have no way to tell. Now, you could enable debug logging, but that is a very invasive operation, requiring you to update the config file on the server, perhaps to restart the server, and isn’t really something that you can just do. This is especially true when we are talking about production system under load, where adding full logging can be very expensive.

You can now set a dynamic logging listener on a running instance, including a production instance:

image

Which then give you a live streaming view of the log:

image

Think about it like doing a tail on a log file, except that this allows you to dynamically configure what logs you are going to watch, and it will only log while you are watching. This is perfect for situations such as “what is this thing doing now?”.

Having access to the log file is great, but it usually have too much information. That is why we also added the ability to get a peek into what requests are actually executing now. This is also a production level feature, which will cause RavenDB to output all the requests so you can see them:

image

This can be very helpful in narrow down “what are the clients asking the server to do”.

Like the production log watch, this is also a feature that is meant for production, so you can subscribe to this information, and you’ll get it. But if there are no subscribers, there is not cost to this.

The HTTP trace feature can be used to watch a single database (which can be very useful on RavenHQ) or all databases (where you’ll need a server admin level of access. To watch the production log, you’ll need to be a server admin.

Tags:

Published at

Originally posted at

Comments (4)

RavenDB 3.0 Days in Sweden

We are going to do a European version of the RavenDB Conf in just over a month, coming to both Malmo and Stockholm for a full day event.

You can see the full details here, but the basic idea is that we are going to be talking about RavenDB 3.0, including showing off all the new stuff, then show a real world use case for managing high scalability systems with RavenDB. We’ll go in depth into the codebase, and then hear about how to make the best use of transformers and indexes and then end the day with a look forward into what has been slowly cooking in our labs and the grand finale with a full guide on how best to build RavenDB applications in RavenDB 3.0

We are actually going to arrive a day early, so if you are located in Malmo, and want us to come to do some on site RavenDB consulting or training on Sep 17, contract us (support@ravendb.net) and we’ll set it up.

You can register to the event using the following link.

Published at

Originally posted at

RavenDB support guarantees

As part of the 3.0 release of RavenDB, we are going to do a remap of our support contracts. We’ll make a formal announcement about it later, but the idea is to offer the following levels:

  • Standard – about $500 a year per serve, business day availability, maximum response within 2 business day.
  • Professional – about $2,000 a year per server, business day availability, maximum response within the same business day.
  • Enterprise – about $6,000 a year per server, 24x7, maximum response within two hours.

In addition to that, we’ll continue to have the community support via the mailing list. That said, I want to make it clear what kind of support guarantees with are giving in the mailing list:

  • None
  • Whatsoever

Put simply, the community mailing list is just that, a way for the community to discuss RavenDB. We look at that, and we try to help, but there is no one assigned to monitor the mailing list, this is pretty much the team waiting for the code to compile or the current test run to complete and deciding to check the mailing list instead of Twitter or the latest Cat Video.

Any support on the mailing list is provided on a ad hoc basis, and should absolutely not be something that you rely on. In particular, people with EMERGENCY or PRODUCTION ISSUE aren’t going to get any special treatment. If you need support, and if you run critical systems, you probably do, you need to purchase that. We provide guarantees and follow through for the commercial support packages.

I’m writing this post after an exchange of words in the mailing list when a user complained that I went offline at 1 AM on a Saturday night and not continue to provide him free support.

Published at

Originally posted at

Comments (9)

Guids are evil nasty little creatures that make me cry

You might have noticed that I don’t like Guids all that much. Guids seems like a great solution when you need to generate an id for something. And then reality intervenes, and you have a non understandable system problem.

Leaving aside the size of the Guid, or the fact that it is not sequential, two pretty major issues with an identifier, the major problem is that it is pretty much opaque for the users.

This was recently thrown in my face again as part of a question in the RavenDB mailing list. Take a look at the following documents. Do you think that those two documents belong to the same category or not?

image

One of the problems that we discovered was that the user was searching for category 4bf58dd8d48988d1c5941735, and the document had category was 4bf58dd8d48988d14e941735. And it drove everyone crazy about how could it be that this wasn’t working.

Here are those Guids again:

  • 4bf58dd8d48988d1c5941735
  • 4bf58dd8d48988d14e941735

Do you see it? I’m going to be putting some visual space and show you the difference.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Here they are:

  • 4bf58dd8d48988d1c5941735
  • 4bf58dd8d48988d14e941735

And if that isn’t enough for you to despise Guids. Feel free to read them to someone else over the phone, or try to find them in a log file. Especially when you have to deal with several of those dastardly things.

I have a cloud machine dedicated to generating and disposing Guids, I hope that in a few thousands years, I can kill them all.

Tags:

Published at

Originally posted at

Comments (27)

RavenDB 3.0 Status Update

We are nearly there. You can see that we are starting to expose all the cool stuff we have done through the past year.

Feature map is currently frozen, and we expect to have a release candidate within 2 weeks and a stable release 4 weeks after that.  There is one big ticket item that still remains, and the rest is just UI work. We’ll probably do the release candidate with an intentionally ugly UI, then roll the new theme for the UI for the actual release, since we don’t want to hold the schedule just for this release.

Things are very exciting here, and we can’t wait for you to take the fruits of our labor for spin. I trust you’ll be impressed Smile.

Tags:

Published at

Originally posted at

Comments (3)

Production analysis and trouble shooting with RavenDB

The annoying thing about software in production is that it is a  black box. It just sits there, doing something, and you have very little input into what. Oh, you can look at the CPU usage and memory consumption, you can try to figure out what is going on from the kind of things that the system will tell you this process is doing. But for the most part ,this is a black box. And not even one that is designed to let you figure out what just happened.

With RavenDB, we have made a very conscious effort to avoid being a black box. There are a lot of end points that you can query to figure out exactly what is going on. And you can use different endpoints to figure out different problems.  But in the end, while that was very easy for us to use, those aren’t really meant for end users. They are meant for our support engineers, mostly. 

We got tired of sending over “give me the output of the following endpoints” deal. We wanted a better story, something that would be easier and more convenient all around .So we sat down and thought about this, and came up with the idea of the Debug Info Package.

image

This deceptively simple tool will capture all of the relevant information from RavenDB into a single zip file that you can mail support. It will also give you a lot of details about the internals of RavenDB at the moment this was produced:

  • Recent HTTP requests
  • Recent logs
  • The database configuration
  • What is currently being indexed?
  • What are the current queries?
  • What tasks are being run?
  • All the database metrics
  • Current status of the pre-fetch queue
  • The database live stats

And if that wasn’t enough, we have the following feature as well:

image

 

We get the full stack of the currently running process!

You can see how this look in full in the here:

stacks

 

But the idea is that we have cracked open the black box, and it is now so much easier to figure out what is going on!

Tags:

Published at

Originally posted at

Comments (9)

RavenDB On Azure

It took a while, but it is here. The most requested feature on the Azure Store is here:

Embedded image permalink

This is currently only available on the East US region. That is going to change, but it will take a bit of time. You can vote on which regions you want RavenHQ on Azure to expand to.

RavenHQ on Azure can be used in one of two ways. You can purchase it via the Azure Marketplace, in which case you have to deal only with a single invoice, and you can manage everything through the Azure site. However, the Azure Marketplace doesn’t currently support prorated and tiered billing, which means that the plans that you purchase in the marketplace have hard limits on data. You could also purchase those same plans directly from RavenHQ and take advantage of usage based billing, which allows you to use more storage than what’s included in the plan at a prorated cost.

RavenHQ is now offering a lower price point for replicated plans, so you don’t have to think twice before jumping into the high availability option.

Tags:

Published at

Originally posted at

Comments (3)

What is my query doing?

Recently we had to deal with several customers support requests about slow queries in RavenDB. Now, just to give you some idea about the scope. We consider a query slow if it takes more than 50ms to execute (excluding client side caching).

In this case, we had gotten reports about queries that took multiple seconds to run. That was strange, and we were able to reproduce this locally, at which point we were hit with a “Duh!” moment. In all cases, the underlying issue wasn’t that the query took a long time to execute, it was that the result of the query was very large. Typical documents were in the multi megabyte ranges, and the query returned scores of those. That means that the actual cost of the query was just transporting the data to the client.

Let us imagine that you have this query:

session.Query<User>()
.Where(x => x.Age >= 21)
.ToList();

And for some reason it is slower than you would like. The first thing to do would probably be to see what is the raw execution times on the server side:
RavenQueryStatistics queryStats;
session.Query<User>()
.Customize(x=>x.ShowTimings())
.Statistics(out queryStats)
.Where(x => x.Age > 15)
.ToList();

Now you have the following information:
  • queryStats.DurationMilliseconds – the server side total query execution time
  • queryStats.TimingsInMilliseconds – the server side query execution time, per each distinct operation
    • Lucene search – the time to query the Lucene index
    • Loading documents – the time to load the relevant documents from disk
    • Transforming results – the time to execute the result transformer, if any
  • queryStats.ResultSize – the uncompressed size of the response from the server

This should give you a good indication on the relative costs.

In most cases, the issue was resolved by the customer by specifying a transformer and selecting just the properties they needed for their use case. That transformed (pun intended) a query that returned 50+ MB to one that returned 67Kb.

Tags:

Published at

Originally posted at

Metrics hunt in RavenDB

You might have noticed that we are paying a lot of options for operational concerns in RavenDB 3.0. This is especially true because we moved away from performance counters to metrics.net, which means that is is much easier and light weight to add metrics to RavenDB.

As a result of that, we are adding a lot of stuff that will be very useful for ops team. From monitoring the duration of queries to the bandwidth available for replication to a host of other stuff.

What I wanted to ask is what kind of things do you want us to track?

Tags:

Published at

Originally posted at

Comments (9)

Small touches: Complex text in RavenDB

This is what we call a “mini feature”, something that you’ll probably not notice unless pointed out to you. Often, we want to store documents that contain multi line strings properties. JSON has a very simple way to handle that:

image

And it works, and if the text is small, it is even readable. But it isn’t really working on anything even remotely complex or long. So we have worked to fix that:

image

Now you can actually read this much more easily. We run into this when we look at stack trace information, where without line breaks, it is nearly impossible to see what is going on.

Tags:

Published at

Originally posted at

Comments (11)

RavenDB Replication Topology Visualizer

Even more goodies are coming in RavenDB 3.0. Below you can see how to visualize the replication topology in a RavenDB Cluster. You can also see that the t5 database is down (marked as red).

image

This is important, since this gives us the ability to check the status of the topology from the point of view of the actual nodes. So a node might be up for one server, but not for the other, and this will show up here.

Beside, it is a cool graphic that you can use in your system documentation and it is much easier to explain Smile.

Tags:

Published at

Originally posted at

Comments (4)

Geo distribution and high availability in RavenDB

A customer asks in the mailing list:

Due to data protection requirements, we have to store a users data closest to where they signed up. For example if I sign up and I’m in London, my data should be stored in the EU.

Given this, how do we ensure when replicating (we will have level 4 redundancy eventually), that any data originally written to a node within say the EU does not get replicated to a node in the states?

The answer here is to use to features of RavenDB together. Sharding and Replication. It is a good thing that they are orthogonal and can work together seamlessly.

Here is how it looks like:

image

The London based user will be sharded to the Ireland server. This server will be replicating to other Ireland based server (or to other servers in the EU). The data never leaves the EU (satisfying the data protection rules), but we get the high availability that we desire.

At the same time, Canadian customers will be served from a nearby states based servers, and they, too, will be replicating to nearby servers.

From a deployment standpoint, what we need to do is the following:

  • Setup a geo distributed sharded cluster using the user’s location.
  • Each shard would be a separate server, which is replicating to N other servers in the nearby geographical area.

And that is pretty much it…

Tags:

Published at

Originally posted at

Comments (6)