Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by email or phone:

ayende@ayende.com

+972 52-548-6969

, @ Q j

Posts: 6,707 | Comments: 48,617

filter by tags archive

Graphs in RavenDB: The overall design

time to read 5 min | 863 words

Note: These series of posts are about a planned feature, exploring how we go about building it. This is meant to solicit feedback and get more eyes on the idea, things aren’t set in stone and we don’t have a firm release date on this.

We have been wanting to add graph queries to RavenDB for several years now, but we always had more important things get in the way. That didn’t prevent us from discussing this internally and sketch up a few options. We are now looking at this more seriously and I thought that sharing the details of our deliberations would be interesting and likely to garner us some valuable feedback. I’m going to assume that the reader is at least somewhat familiar with the notion of graph data and graph queries.

Probably the most well known graph database is Neo4J, which provides the notion of nodes and edges, both of which have a type and a set of (flat) properties. This allow you to define a model of any arbitrary complexity. This works if you model is purely graph based, but it doesn’t work for RavenDB, whose users are used to the document model. On the surface, this looks like a minor detail. RavenDB has documents, which can have any shape, including containing embedded values and collections inside them. Neo4J, on the other hand, model things differently. The simplest example that I can think of is Orders and Order Lines, where you’ll have the following models:

Neo4J

RavenDB

image image

Both models have the same information, but each element in the Neo4J graph is an independent node that is linked to the others. On the other hand, with RavenDB, we have a single document that embeds a lot of the information directly.  Note that what we haven’t shown in the image is that in RavenDB as well, you have other documents as well. The products, for example, are separate documents. 

Graph databases are often used to handle the basis of recommendation engines, fraud detection, etc. But they are usually used to augment the capabilities of the system, rather than as the primary data store of applications. RavenDB, on the other hand, is most frequently deployed as the primary data store. We want to give our users the ability to perform graph operations, but we don’t want to lose anything that make RavenDB useful and easy to use.

We initially thought about having the following definition:

  • Each document is (implicitly) a node in the graph.
  • You can call Link(src,dest,type, attributes) to create an edge between any two documents.
  • Provide the usual graph queries on top of that.

We started exploring this implementation, but it quickly led to mounting complexity. From the point of view of the user, it led to having to do additional work, you’ll have to maintain your document model and the edges at the same time. This allow you to do some interesting things, but it also likely to cause complications down the line and very likely to cause issues when the document model and graph model disagree with one another. Other issues relates to how do you handle graphs in a distributed manner. How do you deal with the creation on an edge between two documents on one node when one of them was deleted on another?

We pushed in that direction for a while, because that was the obvious thing to do, but it really turned up to be a bad idea which didn’t play well with the rest of RavenDB. The worst part was the fact that you might modify the document properties but not define the edge, which lead to inconsistency. This was very easy to do.

The next thing we played with was to remove the Link() call and allow the user to define a background operation that would go and create the links between documents automatically whenever they were updated. This would allow us to avoid having any inconsistencies between the data in the documents and the links between then. After thinking about this for a while, we went ahead with this approach, but removed the requirement for a background operations.

RavenDB will be able to use your existing document model as the graph model as well. In other words, in the model above, you have the orders/2 document, which has two links, for each of the products. This give us both the ability to have a well define document model, with its well known Domain Driven architecture and the ability to hop off all the pre-existing links that we have in the model.

I’ll discuss the querying model and how it all plays together in a future post. For now, I want to show you how this looks like when we want to do a typical graph operation, friends of friends:

image

More details will come in the next post…

RavenDB 4.1, Inside RavenDB and RavenHQ

time to read 3 min | 451 words

imageLast Friday, RavenDB 4.1 hit the RTM milestone and then went out to see what other interesting things it can do. The highlights include:

  1. Cluster wide transactions
  2. JavaScript indexes
  3. SQL Migration Wizard
  4. Distributed Counters
  5. RavenDB Embedded
  6. MongoDB & CosmosDB migration

The Inside RavenDB book is also complete and is now available at Amazon. This ended up being 562 pages of deep dive into everything that RavenDB does, how it does it and most importantly, why it is doing so. Not willing to just give you the raw details, I tried to tell both the story of RavenDB and place it in context, so you’ll understand how to actually make the best use of the database. This has been almost a year and a half in the making. I’m really glad that I wrote the book, because the mere fact that I had to present a coherent story for the book was very helpful for the product itself. Of course, that meant that I had to go and re-write a bunch of stuff, but the end result is both a better product and a better book.

The book is available in both paper back formats and on Kindle (and Kindle Unlimited). I would really appreciate any feedback you have. The actual first copy are currently in the mail (currently residing in Weybridge, Surrey GB) and I can’t wait to actually hold them in my hands.

In other news, we are nearing public availability of 4.1 in RavenHQ as well. RavenHQ is currently in the middle of their private beta for RavenDB 4. Along with supporting v4.1, RavenHQ is revamping it's hosting model to offer features such as hosting multiple databases per subscription and the ability to create a cluster in the region of your choice. RavenHQ will be opening up it's V4.1 clusters to the public within the next few weeks, but the private beta is currently offering clusters for free! If you’d like to participate in the private beta, fill out their survey to get an invitation.

The hosting model revamp sounds small, but it has huge implications on all sort of details that you have to get right when you operate at the scale that RavenHQ does. In particular, whereas the 3.5 offering for RavenHQ is for a single (replicated) database, the RavenDB 4.1 offering is actually a full fledged cluster. This give you the ability to manage your own databases, designate production / test / dev environments, automatically create databases, etc. RavenHQ is still going to operate and manage everything for you, but you will now have far more freedom.

RavenDB 4.1 Tidbits: The production environment

time to read 1 min | 171 words

A few things in RavenDB 4.1 are relatively minor features, but they fill a niche and can really help out. One of them is the server environment reminder. When working with multiple clusters with different environment, it is easy to forget in which environment you are on. That can lead to thinking that you are on a test environment when you are actually on production, with much hilarity down the road for the people reading your post mortem.

The server environment (which you can configure from the Studio Configuration) give you the ability to have a visible reminder where you are at. Here is what this looks like:

image

In some cases, you’ll have a single cluster, but have different databases for different tasks. The environment can be defined at the cluster and database levels, like so:

image

RavenDB 4.1 FeaturesMongoDB & CosmosDB Migration Wizards

time to read 2 min | 345 words

After we built the SQL Migration Wizard for RavenDB 4.1, we started to field questions about assistance in migrating from more databases. As a result of this, we have introduced a support for MongoDB and CosmosDB migration.

I’m going to walk you through how this works. First, you’ll need to download the Release Candidate of RavenDB 4.1. In addition to the zip package, you’ll need to download the Tools zip file as well.

Next, run RavenDB and create a new database, then go to Settings > Import Data and select From other. Here is what this will look like:

image

I went to shodan.io and found a publicly available MongoDB server to test this out. The process completed successfully and gave me a single document:

image

I guess someone already got to this instance.

More seriously, I scanned literally the first page in this listing and was able to connect and retrieve real documents from a few of them. That included what looked like users (hashed) passwords, among other details. I deleted the data

At any rate, you can use this wizard to pull data from a MongoDB instance to your RavenDB database. RavenDB will handle all the details of the transfer for you. There is even the option to use a transformation script to shape the data as it goes into RavenDB.

You can do the same for CosmosDB, as you can see below:

image

These credentials I got from doing a GitHub search.

On the one hand, I’m really happy that the feature works. On the other hand, I’m pretty much in a state of despair from the state of security in general.

We are looking into other databases that our users want to migrate from. If you have such a need, please let us know.

RavenDB 4.1 Release Candidate is out

time to read 1 min | 128 words

Last Friday, without much of a fanfare, we released the Release Candidate of RavenDB 4.1. This release concludes about six months of work to bring some new and awesome features to the table:

  1. Cluster wide transactions
  2. JavaScript indexes
  3. SQL Migration Wizard
  4. Distributed Counters
  5. RavenDB Embedded (for .Net, Python, etc)
  6. MongoDB & CosmosDB migration
  7. Results highlighting
  8. Subscription includes
  9. Explain query plan
  10. Explain query execution
  11. Query timing
  12. Better setup wizard and reducing deployment friction

And these are just the first dozen big features. We made a lot of improvements, supporting wild card certificate generation in the setup process, better spatial units support and a plenty of other stuff that isn’t sexy or attractive on its own but adds up to a major jump in what you can do with RavenDB.

Unexpected use cases for RavenDB in IoT

time to read 3 min | 558 words

imageWe designed RavenDB to be a server side database, to be used to run large scale business applications. Surprisingly for us, there is a large group of users that have taken RavenDB and actually run it as part of their deployed systems. In other words, instead of having a single large RavenDB cluster they will typically deploy many (hundreds in the small cases, tens of thousands to millions in the large cases) of RavenDB instances across a wide variety of locations.

Part of that is the fact that RavenDB can be embedded inside an application quite easily. That means that we don’t need complex setup or administration. You can just use RavenDB from your application and everything Will Just Work. Another factor is the fact that you can run RavenDB on very low end machines, including 32 bits machines, ARM SoC, etc.

One use case was a point of sales system that had to spec out their hardware a decade in advanced and had to deal with existing installations that were still running hardware from 10 years ago (with little desire to upgrade). Another use case was deploying RavenDB as part of an industrial robot package, with RavenDB installed on a 32 bits ARM system on chip that control the robot.

That kind of deployment pattern lead to interesting requests. For example, several of our customers need ad hoc replication in a location. So all the nodes in a particular physical location will join together to a full mesh of replicated nodes. This gives us high availability in a particular location with any node in the network being able to service any request across the entire location. Boot up a new machine, wait a bit for the rest of the network to update it and you are good to go. This also helps when you consider your machines to be unreliable (because they are old, beaten down and generally minimally maintained).

Another scenario with the need for dynamic topologies is the deployment of RavenDB as set of independent nodes that need to report to some sort of head quarters. This is easy to do by defining external replication or ETL on the node and have it send all the relevant data to a central location for processing. This way, you get a cheap “always available” local node but can still have a global view of your data. I posted about something similar in the past, if you care for the details.

We are now looking for additional features to serve this kind of deployment. In particular, we are interested in making it easy to share data and generate analytics across widely distributed and separated set of instances. One of things that we are currently considering is some form of integration with the cloud. For example, consider Amazon Athena, which allow you to run analytics queries on files residing in S3. We can define ETL processes that would upload the data from RavenDB as it is changed on each individual node. This way, you have each node pushing data to the cloud and a central location that can run live analytics on the data.

What are your thoughts on this? And what other features do you think will serve this kind of scenario?

Product Release Postmortem: Things You Should Never Do, Part II

time to read 18 min | 3409 words

imageThis post is the text version of a presentation I gave a few weeks ago. There is in reference of this classic post by Joel.

In 2015, I decided that we needed to reboot RavenDB. I did that with the full understanding that this is going to be a huge task, including knowing that it will be bigger than what I can project, even if I take this line of thinking into account.

RavenDB 1.0 was written a decade ago. It was written because it didn’t leave me alone and I wanted to get it out of my head. At the time, I was focused more on getting it out the door (and my head) and was taking shortcuts in the implementation. That allowed me to cut down dramatically on the amount of work that is involved in it. At the same time, this put some constraints on the implementation and architecture. The most obvious one was the reliance on Esent, which tied us to Windows. C# as the implementation language, to a lesser extent, also had the same issue until .NET Core. (Yes, I’m aware of Mono, I have no idea how people managed to run anything beyond hello world on it. We tried porting RavenDB to Mono multiple times, and I still bear the scars.)

I went back and looked at our release notes, in literally every major release, we have spent significant amount of time and effort on “performance optimizations”. In January of 2015 we had a few sprints that were dedicated to just this issue. We went down to assembly code in some cases, analyzed our hotspots and optimize things in a very serious manner. We got some amazing performance improvements in some cases, reducing the runtime by orders of magnitude in some cases. But it still felt like we were hitting a limit. What is more, experience from customers in production showed us that there were a number of cases where we run into problematic behavior. This mostly happened on large / complex projects. And nearly all those issues were related in one way or another to memory and the GC.

Our indexing, for example, would be reading data from disk into memory. That was meant to save disk I/O during indexing, and including pretty smart prefetching and monitoring behavior. It also had the side effect of loading documents (which can be large) into managed memory and holding on to them long enough to push them into Gen1 and Gen2. Then they would be indexed and need to go away. But given that they were pushed to a higher generation… that meant more expensive collection cycle.

RavenDB was created before the pervasive use of fast disks, and it turns out that in some cases, reading the data from disk was actually faster than parsing it using JSON.Net. In other words, our “I/O bound” process of reading documents was actually dominated by the time it took to parse the JSON text. That does not include the costs of actually cleaning up this memory. Complex JSON documents can have a lot of objects,  and the cost of GC rise with the number of objects that are being tracked. There were pretty fundamental problems, which I didn’t think we could fix in a piecemeal fashion.

That time also coincided with a peak in the number of support incidents that we got. Unlike many other open source projects, we treat support as a cost center, not a revenue center. In other words, we don’t want to have more support, that isn’t how we want to make money. Being a database, we were frequently at the heart of things and our customers and users are very sensitive to any issue that might arise. I’m painting somewhat of a bleak picture, I’m aware. It wasn’t nearly that bad from the point of view of any particular customer. But on aggregate, from our point of view, it felt like a nasty game of whack a mole. As soon as we provided a solution to one customer’s issue, another would pop up, somewhat related but just different enough to not be fixed by the previous change. These weren’t regressions, mind. These were just a lot of places where the changing times violated some of our core assumptions.

Toward the end of 2015, I sat down and really thought about what we needed and were missing. This was the situation as I saw it.

image

There was also the issue that we have learned a lot over the years. We built Voron (our storage engine) from the ground up, we had a lot of experience running in production and we knew what kind of tasks our customers were using us for. I kept thinking that I wished I had a time machine and could do things over properly. Given that my time machine is still in the shop, I decided that we had two options:

  1. Minor fixes along the way – slowly improving our behavior as we stride toward the desired architecture and usage.
  2. Break it all – essentially start from scratch, with a new architecture and write it the way we want it to be written.

The obvious choice was to do this slowly. The problem was that I really couldn’t think of a good way to actually achieve that. The kind of changes we wanted to make started from replacing the most fundamental structure we had, how we represent JSON in our document database and got more complex from there. We wanted to change how we store data on disk, how we index data, how we … literally every single feature that we had was going to be transformed in some way.

We also had additional issues. The Windows only limitation was really hurting us and we really wanted to get a good Linux story going. The support burden was also at the very top of my mind as we considered what to do. In the end, we came up with the following decisions:

  • We don’t require backward compatibility. Either on the server side or client side.
    • That was the hardest decision, but it meant that we could actually tackle some of the biggest issues freely and without constraint.
    • That meant that we wanted to keep the same feeling, but be able to make changes to corners of the API that atrophied.
  • Support cost and simplified operations as a primary concern.
    • This meant that, at the design level, we took into account debugging considerations.
  • Order of magnitude performance improvement across the board.
    • Otherwise, it isn’t worth the effort.
  • Cross platform from the get go.

That was in Sep 2015. I sat down and wrote a design document that outlined the new architectural approach, spiked a few things and then we were off to the races. I blogged all about the process extensively, so I’m not going to repeat that.

We decided to use DNX (which became .NET Core) at a very early stage. Initially, I don’t believe that we even had a debugger, and most of our builds had to be trigger from the command line. I guess that if you are going to make a risky decision, you might as well make a few others…

I’ll say that I made a lot of preparation to fail up front. Part of the reason we went with DNX was that we knew that worst case scenario, we could spend a few days and get it working on the full .NET framework if we had to. I took this step with a lot of backward glances to make sure that we won’t get lost.

Alongside our experience in supporting RavenDB, we also run a UX study and combed all the incident reports we generate from support calls. The idea was to take as much time as necessary to get things as right as we could handle it. The studio change between 3.5 and 4.0 is massive, and was driven by getting a talented professional to design each part of the UI, guided by real world UX study and analysis. We kept asking “where do it hurt?” and whenever we had found a cause of pain we worked to alleviate it.

Some of our guiding principals during that phase of the project were:

  • Cross platform from the get go.
    • We couldn’t afford to port it midway through. Too complex and prune to failure.
  • OWN the stack.
    • We don’t want to use any components that we don’t have good visibility into and the ability to work  with.
    • In particular, anything that is a core competence should be owned and built by us. For our scenario, that means primarily the storage engine.
  • Build for performance.
    • I wasn’t kidding about requiring x10 performance improvement. We had one or two devs at all time running benchmarks and fixing things performance of every completed feature.
  • Build for operations.
    • Each and every design decision should be considered in light of its operational behavior.
    • In particular, we excised any feature that relied on hard to figure out technology or integration (I’m looking at you, Windows Auth).
    • This included changing the design of the software so a core dump would make it easier to figure out what is going on. We also explicitly opened up a lot of the internal behavior as debug endpoints and plug them to the studio so operators will have greater visibility. As an aside, that was very helpful in figuring out our performance bottlenecks and we worked to improve that part of the project as we strived for ever faster performance.
  • Reducing the support burden as an major goal.
    • A lot of the previous points tie into this. But this is also where we combed over any issue that had a “user misconfigured / misused” and built in alerts directly into RavenDB to give the user early warnings about common issues.
  • We defined a set of common scenarios. Reading / writing documents, for example, and then we spent months on designing the whole system so it will work to make these fast, seamless and easy.

A good example of that is how we stored documents in RavenDB now. We have our own binary format that allow us to avoid parsing the document when reading from disk, plays nicely with memory mapped files (which is how Voron, our storage engine, works) and can effectively allow us to hand a pointer to a memory mapped buffer and start working with that as a JSON document without:

  • Allocating any managed memory
  • Parsing JSON
  • Require caching / pre-fetching, etc.

We spent a lot of time thinking about what we want to do, and then we looked into how the operating system expect us to behave. The idea is that if we play to the operating system’s expectation, we can reap a lot of benefits from the OS’ own behavior. This is how RavenDB handles loading data to memory. We let the operating system handle it and just make sure that our own behavior is both predictable and applicable to the OS’ optimizations.

I mentioned that GC was the bane of our existence, right? We moved a lot of the memory management in RavenDB to unmanaged code and handle that explicitly. That gave us the advantage that we know a lot more about how we should expect to use the memory an can spend the time to make this highly optimized.

At the debugger side of things, we made some changes to the design of RavenDB with the intent to make it easier to debug and analyze core dumps. For example, most of the long running threads are named, so it is easy to figure out who they belong to (and not just what they are currently doing). For that matter, long running tasks are using synchronous mode, specifically because it means that we can drop in the debugger / core dump and look at their state. This is much harder to do with async methods. You might have noticed that I mentioned core dumps a few times, right? These are essential to figuring out what is going on with your software on production systems. We learned a lot about production debugging over the years and with RavenDB 4.0 we took steps to make things easier. For example, many data structured in RavenDB have an extra field called tag that is there specifically to provide debugging information about the value if we are looking at the value in the debugger.

An obvious question for this project was whatever we should still stay on .NET or should we move to an unmanaged language. I considered this seriously, with Rust, C/C++ and Go being the top contenders as the implementation language. I decided to stay with .NET for several reasons. Productivity was right there in the top. We already had a team that was well versed in .NET, and while that isn’t a blocker, it was a consideration. The tooling around .NET are leagues ahead of anything else that I have seen. That include both write time (where Rider / ReSharper rules) and for debug time (I found nothing remotely close to Visual Studio for debugging non trivial code easily). The cross platform angle, which was the most serious issue for us, was resolved with .NET Core.

Rust wasn’t matured enough at the time (2015) and even today I think that a language that prides itself in being hard to learn isn’t a good choice. C++ was a strong contender, but the slow compilation times were an issue. The tooling is similar, but inferior in many respects. Cross platform C++ is possible, and modern C++ is very different from what I remember. However, it come with a very high degree of complexity and would take a lot of time to master again properly. C (distinct from C++) is much simpler language. Still has the compilation speed issues, but the language is much simpler. I think that if it had a defer mechanism builtin it would be a much nicer language. Go was ruled out because if I’m going to be writing everything from scratch, I might as well go all the way down to C’s level and not stop with something that still has GC pauses.

The choice of C# as CoreCLR has been vindicated. The project team and the community at large puts a large emphasis on performance and we keep getting more and better way to handle low level details while still able to use higher level concepts when needed. And the tooling… dear God. I routinely work with other platforms, testing things out, but there is nothing that come close to the toolsets that are available for C#.

An interesting wrinkle with the 4.0 release was that we started it before the 3.5 release was even out. For a while, we had a small team working on the foundations of 4.0 while the rest of us were busy hammering in the last details of RavenDB 3.5.

As soon as we had the bare minimum to go (basically, it compiled and could save a single document and even regurgitate it back up again) we started heavy parallelization of the work. We had a team working on indexes while another was dealing with (even at this early stage) performance and another working on the user interface. At that time frame, we have hired a few more people and could really see the benefits of all of the separate teams working in concrete. One of the priorities of this method was to get to a demoable state. In fact, at some point we had over 30% of the people working on either the UI directly or UI related infrastructure.

One of the things we kept hearing back is that the UI and insight it provided into what is going on inside the database were crucial for our users. It also helped a lot to us as we developed RavenDB to get to play with it directly and see things in an easy manner. The UI has been at once one of the most trivial of changes and the most profound. On the one hand, we didn’t really make any significant architectural changes in the UI. On the other hand, we re-wrote most of it with the aid of UX study and a real professional at the helm. That gave us a lot of visible polish that underscore the amount of work that happened in the engine.

How did all of that turn out?

  • I initially thought it would last a year to 15 months, with an expectant due date of Dec 2016. That was with a team size of about 25 people.  Work started in Sep 2015.
  • As it turns out, RavenDB accumulated a lot of features in the years it spent in production. We had to evaluate each of it, see how it would fit into our architecture and get it ported. That took a lot of time. Especially because it many cases we took the time to change the approach we had for the feature completely.
  • By Mid 2016 I already changed the scheduled to Jun 2017.
  • Close to the end of 2016 we released RavenDB 3.5. This freed up some people to work on the 4.0 release, but also meant we had higher than usual support calls while customers integrated the new release.
  • Actual release of the 4.0 release happened in Feb 2018. So just about 30 months from the start or about double the time I expected it to happen.
  • We had to cut some features out to make the 4.0 release, all of them are back in the 4.1 release, scheduled for next month.
    • This means that to get back to the same place took us 3 years. But we now have a lot of extra features.
    • Most of the missing features were pretty minor, though, and rarely used.

What did all of that gain us?

  • Performance: Single node. Over 100,000 writes / sec and over 1,000,000 reads / sec in our benchmarks.
    • Real world users report performance boost of x20 to x52 time faster.
  • Support call duration dropped from days / weeks to about 2 – 4 hours.
  • Cross platform on Windows, Linux, ARM and MacOSX.
    • We are now deployed to production on Raspberry PIs, because we are the fastest real database on that kind of hardware.

We were over a year overdue, and even with the deadline being extended several times we had to cut some features to actually make the cut for release. The general acceptance of the new release by the community has been a roaring success. We exceeded our own goals for the project, even if we took a lot longer than expected to get there.

Now, for some additional thoughts. We didn’t really re-write the whole thing from scratch. Instead, we had a lot of code that we could at least partially reuse. The storage engine was ported, no re-written, for example. However, we changed architectures in a pretty significant way. For example, the format and manner of working with JSON changed entirely between these two released. We are a JSON document database. As you can imagine, we pretty much had to modify everything as a result of that.

We didn’t designed the whole things from the started. We had a rough outline and we let things roll from there. As a result of the new architecture and expectation, by the time we hit a particular feature we were able to utilize what we already learn about how to work with the new architecture to improve things. We also weren’t afraid of changing things multiple times. Authentication had several major design changes midway through, and it ended up so much simpler than what we had before. Even pretty late in the game, we still made significant changes. The RQL support, having a SQL like querying language, came about on the last 20% of the project.

That was a huge change, and I got a lot of “here comes the crazy train again” feedback. This is probably one of the reasons we delayed by another few months. But it was worth it by far. Basically, because we were able to give up on backward compatibility, we were able to move quickly and change stuff as we wished. We knew that we wouldn’t have another change like that for another decade, so we try to get the big changes done.

In retrospect, I think it worked quite well. I’m really proud of how RavenDB 4.0 turned out.

Connection handling and authentication in RavenDB 4.0

time to read 3 min | 531 words

imageAn interesting question has popped up in the mailing list about the behavior of RavenDB. When will RavenDB client send the certificate to the server for authentication? SSL handshake typically takes multiple round trips to negotiate an SSL connection, and that a certificate can be a fairly large object. It makes sense that understanding this aspect of RavenDB behavior is going to be important for users.

In the mailing list, I gave the following answer:

RavenDB doesn’t send the certificate on a per request basis, instead, it send the certificate at the start of each connection.

I was asked for a follow up, because I wasn’t clear to the user. This is a problem, I was answering from my perspective, which is quite different from the way that a RavenDB user from the outside will look at things. Therefor, this post, and hopefully a more complete way of explaining how it all works.

RavenDB uses X509 Client Certificates for authentication, using SSL to both authenticate the remote client to the server (and the server to the client, using PKI) and to ensure that the communication between client and server are private. RavenDB utilizes TLS 1.2 for the actual low level wire transfer protocol. Given that .NET Core doesn’t yet implement TLS 1.3 or FastOpen, that means that we need to do the full negotiation on each connection.

Now, what exactly is a connection in this regard? It this going to be every call to OpenSession? The answer is emphatically not. RavenDB is managing a connection pool internally (actually, we are relying on the HttpClient’s pool to do that). This means that we are only ever going to have as many TCP connections to the server as you had concurrent requests. A session will effectively borrow a connection from the pool whenever it needs to talk to the server.

The connections in the pool are going to be re-used, potentially for a long time. This allow us to alleviate the cost of actually doing the TCP & SSL handshake and amortize it over many requests. This also means that the entire cost of authentication isn’t paid on a per request basis, but per connection. What actually happens is that on the beginning of the connection, the RavenDB server will validate the client certificate and remember what permissions are granted to it. Any and all requests on this connection can then just used the cached permissions for the lifetime of the connection. This stateful approach reduce the overall cost of authentication because we don’t need to run full validation on every request.

This also means that OpenSession, for example, is basically free. All it does is allocate a bunch of dictionaries and some other data structures for the session. There is no wire traffic because the session is created, only when you actually make a request to the server (Load, Query, SaveChanges, etc). Most of the time, we don’t need to create a new connection for that, but can use a pre-existing one from the pool. The entire system was explicitly designed to take advantage of best practices to optimize your overall performance.

RavenDB 4.1 FeaturesThis document is included in your subscription

time to read 2 min | 374 words

Subscriptions in RavenDB allow you to build persistent queries, batch operations and respond immediately to changes in your data. You can read more about them in this post, and I have dedicated a full chapter to discussing them in the book.

In RavenDB 4.1 we improved subscription support by adding the ability to include related documents directly as part of the subscription. Consider the following subscription:

image

The output of this subscription is going to be orders where the geographical coordinates of the subscriptions are not known. We use that to enrich the data by adding the missing location data from the shipping address. This is important for features such as spatial searches, planning deliveries, etc.  For the purpose of this post, we’ll assume that we accept user’s addresses, which do not have spatial information on them and we use a background process to fill them in.

On the one hand, we do want to add the location data as soon as possible but on the other hand, we want to avoid making too many address resolution requests, to avoid having to pay for them. Here is what we come up with to resolve this.

You can see that there are a bunch of interesting things in this code:

  • We can access the related company on the order using Load, and it will not trigger any network call.
  • If the company already has this address, we can use the known location, without having to make an expensive geo-location call.
  • If the company doesn’t have an address, we’ll fill this in for the next run.
  • If the company’s address doesn’t have location, only then we’ll make a remote call to get the actual location data.
  • We don’t call Store() anywhere, because we are using the session instance from the batch, when we call SaveChanges(), all the changes to the loaded documents (either orders or companies) will be saved as a single transaction.
  • Because we update the Location field on the order, we won’t be called again with the updated order, since the subscription filters this.

All in all, this is a pretty neat way to handle the scenario in a very efficient manner.

RavenDB 4.1 FeaturesDetailed query timing details

time to read 1 min | 190 words

Queries are one of the most important things you do in RavenDB, and analyzing queries to understand their costs is an important step in many optimization tasks.

I’m proud to say that for the most part, you don’t need to do this very often with RavenDB. Usually the query optimizer just works and give you query speed that is fast enough that you don’t care how this is achieved. With RavenDB 4.1, we made it even easier to figure out what are the costs of the query. Here is a good example:

image

With the inclusion of “include timings()”, RavenDB will provide you with detailed stats on the costs of the relevant pieces in the query. The output in the studio looks like this:

image

You can see in a glance exactly how much time RavenDB spent on each part of your query. Armed with that information, you can set out to improve things a lot faster (pun intended).

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. Reviewing FASTER (9):
    06 Sep 2018 - Summary
  2. RavenDB 4.1 features (12):
    22 Aug 2018 - MongoDB & CosmosDB Migration Wizards
  3. Reading the NSA’s codebase (7):
    13 Aug 2018 - LemonGraph review–Part VII–Summary
  4. Codex KV (2):
    06 Jun 2018 - Properly generating the file
  5. I WILL have order (3):
    30 May 2018 - How Bleve sorts query results
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats