Ayende @ Rahien

My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by email or phone:


+972 52-548-6969

, @ Q c

Posts: 6,399 | Comments: 47,407

filter by tags archive

Inside RavenDB 4.0 book–Chapter 4 & 5 are done

time to read 1 min | 189 words

imageThe RavenDB 4.0 book is going really well, this week I have managed to write about 20,000 words and the current page count is at 166. At this rate, it is going to turn into a monster in terms of how big it is going to be.

The book so far covers the client API, how to model data in a document database and how to use batch processing in RavenDB with subscriptions. The full drafts are available here, and I would really appreciate any feedback you have.

Next topic is to start talking about clustering and this is going to be real fun.

I’m also looking for a technical editor for the book. Someone who can both tell me that I missed a semi column in a code listing and tell me why my phrasing / spelling / grammar is poor (and likely all three at once and some other stuff that I don’t even know). If you know someone, or better yet, interested and capable, send me a line.

The case of the hanging request in the cache

time to read 2 min | 351 words

imageA user just reported that after a short time of using RavenDB, all requests started to hang and eventually timed out. We are talking about making several dozen requests in the space of a minute or so, as a user operating against a web application.

That was a cause of concern, obviously, and something that was very strange. Luckily, the user was able to send us a stripped down version of his application and we were able to reproduce this state locally. On that particular application only. We couldn’t reproduce this anywhere else.

In fact, we fixed the problem and still don’t know how to reproduce it. The underlying issue is that we are making a GET request using HttpClient and specifying HttpCompletionOption.ResponseHeadersRead in the SendAsync method. Now, the server might return us a 304 response, in which case we’ll use the cached value we have. If we make a few requests like that, all future requests will hang. In order to fix this problem we need to dispose the response, or read the (empty) content of the response.

Now, the crazy thing is that I can probably speculate pretty accurately on why this is going on, we have fixed it, but I can’t reproduce this problem except within the application the user sent us. So no independent reproduction.

The likely issue is that the HttpClient considers the request open until we read all its content (even if there is non), and that as we make more requests that have the same 304 and therefor no access to the content of the request, we have more and more requests that are considered to be open, up to the maximum number of requests that are allowed against the server concurrently. At this point, we are waiting for timeout, and it look like we are hanging.

All very reasonable and perfectly understandable (if annoying because you only need to dispose it sometimes is a trap waiting to happen), but I don’t know why we couldn’t reproduce it.

RavenDB 4.0The etag simplification

time to read 2 min | 357 words

A seemingly small change in RavenDB 4.0 is the way we implement the etag. In RavenDB 3.x and previous we used a 128 bits number, that was divided into 8 bits of type, 56 bits of restarts counter and 64 bits of changes within the current restart period. Visually, this looks like a GUID:  01000000-0000-0018-0000-000000000002.

The advantage of this format is that it is always increasing, very cheap to handle and requires very little persistent data. The disadvantage is that it is very big, not very human readable and the fact that the number of changes reset on every restart means that you can’t make meaningful deduction about relative sizes between any two etags.

In RavenDB 4.0 we shifted to use a single 64 bits number for all etag calculations. That means that we can just expose a long (no need for the Etag class) which is more natural for most usages. This decision also means that we need to store a lot less information, and etags are one of those things that we go over a lot.  A really nice side affect which was totally planned is that we can now take two etags and subtract them and get a pretty good idea bout the range that needs to be traversed.

Another important decision is that everything uses the same etag range. So documents, revisions, attachments and everything share the same etag, which make it very simple to scan through and find the relevant item just based on a single number. This make it very easy to implement replication, for example, because the wire protocol and persistence format remain the same.

I haven’t thought to write about this, seemed like too small a topic for post, but there was some interest about it in the mailing list, and enumerating all the reasons, it suddenly seems like it isn’t such a small topic.

Update: I forgot to mention, a really important factor of this decision is that we can do do this:


So we can give detailed information and expected timeframes easily.

RavenDB 4.0Data subscriptions, Part II

time to read 3 min | 493 words

imageIn my previous post I introduced data subscription and mentioned that there is more to it than just being able to get push based reliable stream of documents as they change. The problem is that RavenDB will send us the current document, and if the document has been modified multiple times quickly enough, we’ll only get it once. What is more, we are getting the document in our client code, but all we know is that it was changed, no what changed.

With RavenDB 4.0 we now have versioned subscriptions, working alongside the versioning feature. First, we define that a particular collection will have versioning enabled:


And now we can make use of versioned subscriptions.

In this case, you can see that we make use of Versioned<User> type, which indicates to RavenDB that we are interested in a versioned subscription. Instead of sending us just the modified document, we’ll get both the Previous and Current version of the document. In fact, we’ll be called with the Previous / Current version of the document on each change. You might have noticed the null checks in the subscription code, this is because when a document is created, we’ll get it with null Previous value and when a document is deleted, we’ll get it with a null Current value. If the document has been deleted and recreated, we’ll be called first with a Previous instance and null Current  and then null Previous and a Current instance.

In other words, you are now able to track the entire history of a document inside RavenDB, and make decisions based on that. Like regular subscriptions, we have the ability to script a lot of the logic, like so:

What this subscription will do is to analyze all the changes on a user, and then send us the user document as it was banned.

It is important to note that this doesn’t require you to be monitoring the subscription as it happens, you can do this at any point, and you’ll get the historical data. For that matter, this is also a high available solution. If a client goes down, it (or another client) can resume from where it left off, and if the server goes down, the client will transparently be able to failover to a replica without any user or admin involvement, running from where it left off.

We only started looking into the implications of this feature, but the potential for analytics on the changes is already quite obvious. We are going to send you the data in the order it was generated, so you can build a model of changes as it make sense in your domain, without having to go to the trouble of manually keeping track of everything.

RavenDB 4.0Data subscriptions, Part I

time to read 3 min | 516 words

imageI’ll be talking about this feature more once the UI for it is complete, but this feature just landed in our v4.0 branch and it is so awesome that I can’t help talking about it right away.

In RavenDB 4.0 we have taken the idea of subscriptions and pushed it up a few notches. Data subscriptions gives you a reliable push based method to get documents from RavenDB. You setup a subscriptions, and then you open it and RavenDB will stream to you all the documents that are relevant to your subscription. New documents will be sent immediately, and failures are handled and retried automatically. Subscriptions are a great way to build all sort of background jobs.

In RavenDB 3.x their main strength was that they gave you a reliable push base stream of documents, but in RavenDB 4.0, we decided that we want more. Let us take it in stages, here is the most basic subscription usage I can think of:

This is subscribing to all User documents, and RavenDB will first go through all the User documents, sending them to us, and then keep the connection alive and send us the document whenever a User document is updated. Note that we aren’t talking about just a one time thing. If I modify a document once an hour, I’ll be getting a notification on each change. That allows us to do hook this up to jobs, analytics, etc.

The really fun thing here is that this is resilient to failure. If the client maintaining the subscription goes down, it can reconnect and resume from where it left off. Or another client can take over the subscription and continue processing the documents. In RavenDB 4.0, we now also have high availability subscriptions. That means that if a server goes down, the client will simply reconnect to a sibling node and continue operating normally, with no interruption in service.

But you aren’t limited to just blinding getting all the documents in a collection. You can apply a filter, like so:

In this manner, we’ll now only get notified about active users, not all of them. This filtering allows you to handle some really complex scenarios. If you want to apply logic to the stream of changed documents, you can, getting back only the documents that match whatever logic you have in your document.

But the script can do more than just filter, it can also transform. Let us say that we want to get all the active users, but we don’t need the full document (which may be pretty big), we just want a few fields from it.

In this manner, you can select just the right documents, and just the right values you need from the document and process them in your subscription code.

There is actually more, which I’ll post in the next post, but I’m so excited about this feature that I’m not even going to wait for the next publishing schedule and push this immediately. You can read it here.

Inside RavenDB 4.0: Chapter 3 is done

time to read 1 min | 96 words

imageI have just completed writing the 3rd chapter for Inside RavenDB 4.0, and the full book is available here (all three chapters of it). It is just under 100 pages in size, and I’m afraid that at this rate this is going to be a massive tome.

The content so far covers setting up RavenDB 4.0 and doing basic CRUD operations, then goes on to deal with modeling documents inside RavenDB.

Any and all feedback is welcome.

RavenDB 4.0 Licensing & Pricing

time to read 4 min | 772 words

Let us start with the biggest news. We are offering RavenDB 4.0 FREE for production usage for single node deployments.

In addition to that, the license for the RavenDB Client APIs is going to change from AGPL to MIT. We currently have clients for .NET, JVM, Python and Node.JS, with Go and Ruby clients coming (hopefully by the time we hit RC). All of which will be available under the MIT license. The server license will remain AGPL / Commercial, with a Community deployment option that will come free of charge. The Community Edition is fully functional on a single node.

You can use this edition to deploy production systems without needing to purchase a license, completely FREE and unencumbered.

We have also decided to do somewhat of a shakeup in the way we split features among editions, primarily by moving features down the slide, so features that used to be Enterprise only are now more widely available. The commercial edition will be available in either Professional and Enterprise editions. For our current subscribers, we are very happy that you’ve been with us for all this time, and we want to assure you that as long as your subscription will be valid, the pricing for it will stay as is.

Both Professional and Enterprise Editions are going to offer clustering, offsite backups, replication and ETL processes to SQL and RavenDB databases. The Enterprise edition also offers full database encryption, automatic division of work between the nodes in the cluster (including failover & automatic recovery), snapshot backups and SNMP integration among other things. Commercial support (including 24x7 production) will be available for Professional and Enterprise Editions.

Since I’m pretty sure that the thing that you are most interested in is pricing information and feature matrix, so here is the information in an easy to digest form.







$749 per core*

$1,319 per core*


Up to 4

Up to 32



Up to 6 GB

Up to 64 GB


Cluster Size

Single node

Up to 5



Community only








Local full backups

Full & incremental
Local / remote / cloud

Full & incremental
Local / remote / cloud

Full database snapshot1




Tasks distribution2

(backups, ETL, subscriptions)

Single node



Highly available tasks3




Highly available databases &
automatic failover




ETL Support


SQL & RavenDB

SQL & RavenDB

Full database encryption








Client authentication via




* To save you the math, a 6 cores server with Professional edition will be about 4,494$ and Enterprise edition will be about 7,914$.

  1. Snapshots capture the database state and allow to reduce restore time on large databases, at the expense of disk space. Snapshots can work together with incremental backup to get point in time recovery.
  2. Tasks (such as backups, ETL processes, subscriptions, updating new nodes, etc) are assigned to specific node dynamically, to spread load fairly in the cluster.
  3. When a node responsible for a task goes down, the cluster will automatically re-assign that task without interruption in service.

Task distribution and failover is best explained via a concrete example. We have a database with 3 nodes, and we define an ETL process to send some of the data to a reporting database. That work will be dynamically assigned to a node in the cluster, and balanced with all other work that the cluster need to do. For Enterprise Edition, if the node that was assigned that task failed, the cluster will automatically transfer all such tasks to a new node until the node will recover.

The new model comes with significant performance boost, all the features that I mentioned earlier and multi-platform support. But we are not forgetting about newcomers, small enterprises and individual clients. For those we have introduced a Community version, a completely FREE license that should suit their needs.

Again, for all existing subscribers, we assure you that while your subscription is valid, its pricing will stay as is. In fact, given that we will grandfather all existing subscriptions at the current price point, and you can purchase a subscription now, before the official release of 4.0, you have a nice arbitrage option available now.

The beta release represent over two years of work by the RavenDB Core Team to bring you top of the line database that is fast, safe and easy to use. It is chockfull of features, to the point where I don’t know where to start blogging about some of the cool stuff that we do (don’t worry, those posts are coming).

RavenDB 4.0 Beta1 is now available

time to read 4 min | 671 words

imageI’m really proud to announce that we have released the RavenDB 4.0 beta.  It has been a bit over six months of so since the alpha release (I can’t believe it has been that long), and we have been working hard on completing the feature set that we want to have for the final release. We are now ready to unveil RavenDB 4.0 and show you what we can do with it.

Since the alpha release, we completed the refactoring to the clustering infrastructure, so a RavenDB node is always in a cluster (even if just a single one), added attachments, full database encryption and subscriptions, improved indexing performance and performance in general and had done a lot of work to make the database more accessible and easier to use. Cluster scenarios are easier and more robust, you can now take a database and span it on multiple nodes at will and you get the usual RavenDB safety, reliability and fault tolerance.

Users coming from the RavenDB 3.x version will notice (immediately after the studio theme) that everything is much faster. Our internal testing shows anything between 10x to 100x improvement in speed over the previous version.

RavenDB is now capable of handling over 100K req/second for writes (that is over hundred thousands requests per second), and much more for reads, with the  caveat that we always hit the network capacity before we hit RavenDB’s capacity. That is per node, so you can scale the number of nodes in the cluster, so can you scale the amount of reads and writes you can handle.

RavenDB 4.0 also comes with much smarter query optimizer, allowing it to generate optimal queries for aggregation in addition to simple queries. And map/reduce in general has been worked heavily to make it much faster and more responsive.

You can download the RavenDB 4.0 beta from our download page, and NuGet package is available by running:

Install-Package RavenDB.Client -Version 4.0.0-beta-40014 -Source https://www.myget.org/F/ravendb/api/v3/index.json

You can run the beta on Windows, Linux, Raspberry PI and Docker, and you can access the live test instance to check it out.

Known issues include:

  • Identities aren’t exported / imported
  • Cluster operations sometimes stall and timeout.
  • The studio currently assumes that the user is administrator.
  • Highlighting & spatial searches aren’t supported.
  • Deployment in hardened environment is awkward.
  • Upgrading from previous versions is only possible via smuggler.
  • Custom analyzers are not currently supported.

If you are upgrading from 3.x or the 4.0 Alpha, you’ll need to export the data and import it again. Automatic upgrades will be part of the RC.

Please remember, this is a beta. I included some of the known issues in this post to make sure that you remember that. We expect users to start developing with RavenDB 4.0 from the beta, and API and behavior are pretty fixed now. But you shouldn’t be deploying to production with this, and you should be aware that upgrading from the beta to RC is going to be done using smuggler as well.

We’ll likely have a few beta releases on the road to RC, fixing all the issues that will pop up along the way. Your feedback is crucial at this stage, since subjecting RavenDB to real world conditions is the only way to battle test it.

In the studio, you can use the feedback icon to send us your immediate feedback, and there is also the issue tracker.


Take RavenDB for a spin, setup a node, setup a cluster, see how it all works together. If we did our job right, you should be able to figure out everything on your own, which is good, because the docs are still TBD. Feedback on bugs and issues is important, but I’m also very interested in feedback on the flow. How easy it is to do things, deploy, setup, connect from the client, etc.

Distributed work in RavenDB 4.0

time to read 4 min | 660 words

imageI talked about the new clustering mode in RavenDB 4.0 a few days ago. I realized shortly afterward that I didn’t explain a crucial factor. RavenDB has several layers of distributed work.

At the node level, all nodes are (always) part of a cluster. In some cases, it may be a cluster that is a single node, but in all cases, a node is part of a cluster. Cluster form a consensus between all the nodes (using Raft) and all cluster wide operations go through Raft.

Cluster wide operations are things like creating a new database, or assigning a new node to the database and other things that are obviously cluster wide related. But a lot of other operations are also handled in this manner. Creating an index goes through Raft, for example. And so does high availability subscriptions and backup information. The idea is that the cluster holds the state of its databases, and all such state flow through Raft.

A database can reside in multiple nodes, and we typically call that a database group (to distinguish from the cluster as a whole). Data written to the database does not go out over Raft. Instead, we use multi master distributed mesh to replication all data (documents, attachments, revisions, etc) between the different nodes in the database. Why is that?

The logic that guides us is simple. Cluster wide operations happen a lot less often and require a lot more resiliency to operate properly. In particular, not doing consensus resulted in having to deal with potential conflicting changes, which was a PITA. On the other hand, common operations such as document writes tend to have a lot more stringent latency requirements, and what is more, we want to be able to accept writes even in the presence of failure. Consider a network split in a 3 nodes cluster, even though we cannot make modifications to the cluster state on the side with the single node, we are still able to accept and process write and read requests. When the split heals, we can merge all the changes between the nodes, potentially generating (and resolving) conflicts as needed.

The basic idea is that for data that is stored in the database, we will always accept the write, because it it too important to let it just go poof. But for data about the database, we will ensure that we have a consensus for it, since almost all such operations are admin based and repeatable.

Those two modes end up creating an interesting environment. At the admin level, they can work with the cluster and be sure that their changes are properly applied cluster wide. At the database level, each node will always accept writes to the database and distribute them across the cluster in a multi master fashion. A client can choose to accept a write to a single node or a to a particular number of nodes before considering a write successful, but even with network splits, we can still remain up and functioning.

A database group has multiple nodes, and all of them are setup to replicate to one another in master/master setup as well as distribute whatever work is required of the database group (backups, ETL, etc). What about master/slave setups?

We have the notion of adding an outgoing only connection to a database group, one of the nodes in the cluster will take ownership on that connection and replicate all data to it. That allow you to get master/slave, but we’ll not failover to the other node, only to a node inside our own database group. Typical reasons for such a scenario is if you want to have a remote offsite node, or if you have the need to run complex / expensive operations on the system and you want to split that work away from the database group entirely.

Artificial documents in RavenDB 4.0

time to read 2 min | 228 words

Artificial documents are a really interesting feature. They allow you to define an index, and specify that the result of the index will be… documents as well.

Let us consider the following index, running on the Norhtwind dataset.


We can ask RavenDB to output the result of this index to a collection, in addition to the normal indexing. This is done in the following manner:


And you can see the result here:


The question here is, what is the point? Don’t we already have the exact same data indexed and available as the result of the map/reduce index? Why store it twice?

The answer is quite simple, with the output of the index going into documents, we can now define additional indexes on top of them, which give us the option to very easily create recursive map/reduce operations. So you can do daily/monthly/yearly summaries very cheaply. We can also apply all the usual operations on documents (subscriptions and ETL processes come to mind immediately). That give you a lot of power, and without incurring a high complexity overhead.


  1. Zombies vs. Ghosts: The great debate - 12 hours from now
  2. Bug stories: The data corruption in the cluster - about one day from now
  3. Bug stories: How do I call myself? - 3 days from now
  4. Bug stories: The memory ownership in the timeout - 4 days from now
  5. We won’t be fixing this race condition - 5 days from now

And 2 more posts are pending...

There are posts all the way to Jul 04, 2017


  1. RavenDB 4.0 (8):
    13 Jun 2017 - The etag simplification
  2. PR Review (2):
    23 Jun 2017 - avoid too many parameters
  3. Reviewing Noise Search Engine (4):
    20 Jun 2017 - Summary
  4. De-virtualization in CoreCLR (2):
    01 May 2017 - Part II
View all series



Main feed Feed Stats
Comments feed   Comments Feed Stats