Ayende @ Rahien

Oren Eini aka Ayende Rahien CEO of Hibernating Rhinos LTD, which develops RavenDB, a NoSQL Open Source Document Database.

You can reach me by:


+972 52-548-6969

Posts: 6,967 | Comments: 49,573

filter by tags archive
time to read 1 min | 111 words

I posted about our RavenDB C++ client a while ago, but I was really bad about making sure that we have regular updates. We have actually finished it already, there are even articles about it available now.

We had a very simple challenge in building our C++ client. We want to give you the same level of comfort and features set in C++ as you would get in a managed language. While keeping the same level of performance you’ll expect from a C++ application. I think we have done so quite successfully. You can read the article for the full details. No gore included Smile.

time to read 1 min | 164 words

I’ll be talking about Building a Grownup Database in the Big Data and Cloud Meetup in Santa Clara, March 18.

I’m going to show some of the features (and the thinking behind them) that went into making RavenDB a simpler database to develop against and operate.


A database is a complex, often fussy beast. For years, Oren Eini has made his living by fixing performance issues of various kinds. After seeing the same mistakes happen again and again, Oren decided to build his own database where these problems will never arise. RavenDB (https://ravendb.net/) started as a solution to the universal problems with relational models, and has been deployed in production for over a decade.
Oren Eini will talk about the kind of features that make RavenDB a grown up database:
-- It doesn't need a full-time babysitter
-- Uses AI automatic indexing and self optimizing engines
-- Understands the operational environment and adjusts to it without the need for a human in the loop
-- High Availability
-- Secured development

time to read 3 min | 434 words

RavenDB makes extensive use of certificates for authentication and encryption. They allow us to safely communicate between distributed instances without worrying about a man in the middle or eavesdroppers. Given the choices we had to implement authentication, I’m really happy with the results of choosing certificates as the foundation of our authentication infrastructure.

It would be too good, however, to expect to have no issues with certificates. The topic of this point is a puzzler. A user has chosen to use a self signed certificate for the nodes in the cluster, but was unable to authenticate between the servers unless they registered the certificate in the OS’ store.

That sounds reasonable, right? If this is a self signed certificate, we obviously don’t trust it, so we need this extra step to ensure that we do trust it. However, we designed RavenDB specifically to avoid this step. If you are using a self signed certificate, the server will trust its own certificate, and thus will trust anyone that is using the same certificate.

In this case, however, that wasn’t happening. For some reason, the code path that we use to ensure that we trust our own certificate was not being activated, and that was a puzzler indeed.

One of the things that RavenDB does on first startup is to try to connect to itself as a client. It checks whatever it is successful or not. If not, we’ll try again, ignoring the registered root CAs. If we are successful at that point, we know what the issue here and ensure that we ignore the untrusted signer on the certificate. We only enable this code path if by default we don’t trust our own certificate.

Looking at the logs, we could see that we got a failure when talking to ourselves, some sort of a device not ready issue. That was strange. We hooked strace to look into what was going on, but there was nothing that was wrong at the sys call level. Then we looked into what was going on and realized that the issue was that the server’s was configured to use: https://ravendb-1.francecentral.cloudapp.azure.com/ but was actually hosted on https://ravendb-1-tst.francecentral.cloudapp.azure.com/

Do you see the difference?

The server was try to contact itself using the configured hostname. It failed, because of a DNS issue, so it couldn’t contact itself to figure out that the certificate was invalid. At that point, it didn’t install the hook and wouldn’t trust the self signed certificate.

So the issue started with investigating why we nodes in the cluster don’t trust each other with self signed certificate and got resolved by a simple configuration error.

time to read 2 min | 281 words

Subscriptions in RavenDB gives you a great way to handle backend business processing. You can register a query and get notified whenever a document that matches your query is changed. This works if the document actually exists, but what happens if you want to handle a business process relating to document’s deletion ?

I want to explicitly call out that I’m generally against deletion. There are very few business cases for it. But sometimes you got to (GDPR comes to mind) or you have an actual business reason for this.

A key property of deletion is that the data is gone, so how can you process deletions? A subscription will let you know when a document changes, but not when it is gone. Luckily, there is a nice way to handle this. First, you need to enable revisions on the collection in question, like so:


At this point, RavenDB will create revisions for all changed documents, and a revision is created for deletions as well. You can see deleted documents in the Revisions Bin in the Studio, to track deleted documents.


But how does this work with Subscriptions? If you’ll try to run a subscription query at this point, you’ll not find this employee. For that, you have to use versioned subscription, like so:


And now you can subscribe to get notified whenever an employee is deleted.

time to read 2 min | 260 words

These are not the droids you are looking for! – Obi-Wan Kenobi

Sometimes you need to find a set of documents not because of their own properties, but based on a related document. A good example may be needing to find all employees that blue Nissan car. Here is the actual model:


In SQL, we’ll want a query that goes like this:

This is something that you cannot express directly in RavenDB or RQL. Luckily, you aren’t going to be stuck, RavenDB has a couple of options for this. The first, and the most closely related to the SQL option is to use a graph query. That is how you will typically query over relationships in RavenDB. Here is what this looks like:

Of course, if you have a lot of matches here, you will probably want to do things in a more efficient manner. RavenDB allows you to do so using indexes. Here is what the index looks like:

The advantage here is that you can now query on the index in a very simple manner:

RavenDB will ensure that you get the right results, and changing the Car’s color will automatically update the index’s value.

The choice between these two comes down to frequency of change and how large the work is expected to be. The index favors more upfront work for faster query times while the graph query option is more flexible but requires RavenDB to do more on each query.

time to read 2 min | 254 words

We run a lot of benchmarks internally and sometimes it feels like there is a roaming band of performance focused optimizers that go through the office and try to find under utilized machines. Some people mine bitcoin for fun, in our office, we benchmark RavenDB and try to see if we can either break a record or break RavenDB.

Recently a new machine was… repurposed to serve as a benchmarking server. You can call it a right of passage for most new machines here, I would say. The problem with that machine is that the client would error. Not only would it fail, but at the exact same interval. We tested that from multiple clients and from multiple machines and found that every 30 minutes on the dot, we’ll have an outage that lasted under one second.

Today I come to the office to news that the problem was found:


It seems that after 30 minutes of idle time (no user logged in), the machine would turn off the ethernet, regardless of if there are active connections going on. Shortly afterward it would be woken up, of course, but it would be down just enough time for us to notice it.

In fact, I’m really happy that we got an error. I would hate to try to figure out latency spikes because of something like this, and I still don’t know how the team found the root cause.

time to read 2 min | 311 words

RavenDB always had optimistic concurrency, I consider this to be an important feature for building correct distributed and concurrent systems. However, RavenDB doesn’t implement pessimistic locking. At least, not explicitly. It turns out that we have all the components in place to support it. If you want to read more about what pessimistic locking actually is, this Stack Overflow answer has good coverage of the topic.

There are two types of pessimistic locking. Offline and online locking. In the online mode, the database server will take an actual lock when modifying a record. That model works for a conversation pattern with the database. Where you open a transaction and hold it open while you mutate the data. In today’s world, where most processing is handled using request / response  (REST, RPC, etc), that kind of interaction is rare. Instead, you’ll typically want to use offline pessimistic lock. That is, a lock that can live longer than a single transaction. With RavenDB, we build this feature on top of the usual optimistic concurrency as well as the document expiration feature.

Let’s take the classic example of pessimistic locking. Reserving seats for a show. Once you have selected a seat, you have 15 minutes to complete the order, otherwise the seats will automatically be released. Here is the code to do this:

The key here is that we rely on the @expires feature to remove the seatLock document automatically. We use a well known document id to coordinate concurrent requests that try to get the same seat. The rest is just the usual RavenDB’s optimistic concurrency behavior.

You have 15 minutes before the expiration and then it goes poof. From the point of view of implementing this feature, you’ll spend most of your time writing the edge cases, because from the point of view of RavenDB, there is really not much here at all.

time to read 3 min | 568 words

We are now working on proper modeling scenarios for RavenDB’s time series as part of our release cycle. We are trying to consider as many possible scenarios and see that we have good answer to them. As part of this, we looked at applying timeseries in RavenDB to problems that were raised by customers in the past.

The scenario in question is storing data from a traffic camera. The idea is that we have a traffic camera that will report [Time, Car License Number, Speed] for each car that it capture. The camera will report all cars, not just those that are speeding. Obviously, we don’t want to store a document for each and every car registered by the camera. At the same time, we are interested in knowing the speed on the roads over time.

There for, we are going to handle this in the following manner:

This allows us to handle both the ticket issuance and recording the traffic on the road over time. This works great, but it does leave one thing undone. How do I correlate the measurement to the ticket?

In this case, let’s assume that I have some additional information about the measurement that I record in the time series (for example, the confidence level of the camera in its speed report) and that I need to be able to go from the ticket to the actual measurement and vice versa.

The question is how to do this? The whole point of time series is that we are able to compress the data we record significantly. We use about 4 bits per entry, and that is before we apply actual compression here. That means that if we want to be able to use the minimal amount of disk space, we need to consider how to do this.

One way of handling this is to first create the ticket and attach the Ticket’s Id to the measurement. That is where the tag on the entry comes into play. This works, but it isn’t ideal. The idea about the tag on the entry is that we expect there to be a lot of common values. For example, if we have a camera that uses two separate sensors, we’ll use the tag to denote which sensor took the measurement. Or maybe it will use the make & model of the sensor, etc. The set of values for the tag is expected to be small and to highly repeat itself. If the number of tickets issued is very small, of course, we probably wouldn’t mind. But let’s assume that we can’t make that determination.

So we need to correlate the measurement to the ticket, and the simplest way to handle that is to record the time of the measurement in the ticket, as well as which camera generated the report. With this information, you can load the relevant measurement easily enough. But there is one thing to consider. RavenDB’s timestamps use millisecond accuracy, while .NET’s DateTime has 100 nanosecond accuracy. You’ll need to account for that when you store the value.

With that in place, you can do all sort of interesting things. For example, consider the following query.

This will allow us to show the ticket as well as the road conditions around the time of the ticket. You can use it to say “but everyone does it”, which I am assured is a valid legal defense strategy.

time to read 3 min | 557 words

One of the first features that RavenDB had, from the very first release, was multi document ACID transactions. With RavenDB you could modify multiple documents at the same time and then save them all, knowing that they would be be saved as a single atomic unit. Other NoSQL databases had you jump through fire if you wanted transactional behavior (building your own two phase commit protocol at the application level, not fun). I consider this to be a pretty important feature, obviously. But why?

In Inside RavenDB book, there is the following advice about modeling considerations for documents:

  • Independent, meaning a document should have its own separate existence from any other documents.
  • Isolated, meaning a document can change independently from other documents.
  • Coherent, meaning a document should be legible on its own, without referencing other documents.

In other words, with proper modeling, you shouldn’t need to have multi document transactions. Any transaction should only contain a single document, so that should be enough, no? Why spend all the time and effort on building multi document transactions?

Well, to start with “proper modeling” is a very loaded term. It is great if you can get it, but there are many cases where you need to deviate from it for various reasons. For example, in this blog, we represent a blog post as two documents. One holds the text of the post, the other holds the comments for the post. The reason for this separation is that there are many cases where you want only the blog post, and not the (potentially very many) comments.

In this case, the layout of the document is subject to the physical realities. It is better to split the document to multiple documents based on their purpose. However, if I add a comment to a post, I want to both add it to the Comments document and to increase the NumberOfComments property on the Post document. Doing this in a single transaction means that I don’t have to worry about consistency.

Another good example of wanting to work with multiple documents at the same time is when my documents aren’t using just holding data. Consider the case of accepting a new employee to the company. I need to:

  • Create the new Employee document.
  • Create initial workflow requirements (orientation, employee handbook, tax papers, setup machine / user / vpn access, etc).

In other words, each document here is independent, but they are created together. I want to create the new Employee’s document and at the same time setup a task for the IT to create a new user, allocate a machine, etc. I want to have the employee go through orientation, do all the require paper work, etc.

Each one of those is modeled as a separate document, because they are. See the definition above and consider how they match. But I absolutely don’t want to have a partial state. That I have a new employee, but I didn’t setup payroll for them is a big problem. Having multi document transaction make things a lot simpler.

You can argue that it would be better to model things as a set of related services with independent databases. I’m not sure that I disagree, but this is a much more complex architecture. Not having to go there on day one, while having clear and easy to use consistency guarantees is a major plus in my eyes.


No future posts left, oh my!


  1. Production postmortem (28):
    21 Feb 2020 - The self signed certificate that couldn’t
  2. RavenDB 5.0 (2):
    21 Jan 2020 - Exploring Time Series–Part II
  3. Webinar (2):
    15 Jan 2020 - RavenDB’s unique features
  4. Challenges (2):
    03 Jan 2020 - Spot the bug in the stream–answer
  5. Challenge (55):
    02 Jan 2020 - Spot the bug in the stream
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats