RavenDB 5.3 New FeaturesTCP Compression
Most of the time, you’ll communicate with RavenDB using HTTP, making REST calls. When you are doing that, you can take advantage of request compression. If the client indicates that it is able to by sending a Content-Encoding: gzip, RavenDB will send the data to you compressed. Given that we are working with JSON texts, which compress very well, we are looking at pretty significant savings in network bandwidth. This has been the case for RavenDB for many years (I didn’t check, but at least a decade, I believe).
There are certain cases, however, where RavenDB will use a binary protocol instead of HTTP. Those are usually scenarios where we are communicating directly with another RavenDB instance. All internal communications between RavenDB nodes will use direct TCP connections and when using Subscriptions, the client will open a TCP connection for the server and use that on a long term basis.
One of the fallacies of distributed computing is that bandwidth is infinite. One of the realities of cloud computing, on the other hand, is that you are paying for bandwidth. Even when you are running inside the same cloud region, cross availability zone network traffic is still charged. As you can imagine, on active systems, you may notice that you are spending a lot of bandwidth on inter cluster communication.
With RavenDB 5.3, we have added compression support for the replication and subscription connections . That means that replication and subscriptions will default for compressing the data. We are using the Zstd algorithm. In our tests, it produced both a higher compression ratio and faster performance than GZip. You don’t have to do anything for this to work (although there is a configuration option "Server.Tcp.Compression.Disable" to disable that if you really want to). When you upgrade to RavenDB 5.3, the cluster will automatically start compressing all traffic.
In our tests, we are seeing 85% (!) reduction in the amount of network traffic that we send out. That is something that I’m very much looking to seeing in our metrics once this is rolled out completely.
This is a RavenDB 5.3 feature (expected mid November) and will be available in the Professional and Enterprise editions of RavenDB.
More posts in "RavenDB 5.3 New Features" series:
- (26 Nov 2021) Revisions includes
- (25 Nov 2021) JSON Patch
- (24 Nov 2021) Studio & Query improvements
- (23 Nov 2021) TCP Compression
- (18 Nov 2021) Experimental PostgreSQL wire protocol
- (17 Nov 2021) Elasticsearch ETL
- (15 Nov 2021) Incremental time series & implementing lambda based accounting
- (12 Nov 2021) Incremental time series
- (11 Nov 2021) Concurrent Subscriptions & Serial operations
- (10 Nov 2021) Concurrent subscriptions
Comments
I assume that compression of SSL connections (whether binary or https) is not supported, due to CRIME or BREACH?
Sebastiaan,
We are using TLS + compression, yes. In general, CRIME & BREACH requires you to be able to:* Write data you control to the source* Observe the encrypted connection However, with RavenDB, there are a bunch of things here that change how this works. First, we are talking about continuous connections, not just a single request you can observe.
Other writes to the database will also be affecting the connection as well.
More to the point, RavenDB already applies several transformations & compression to the plain text data before it is sent over the wire. Taken together, we believe that the overall benefit is far greater than the cost potential risk.
Comment preview