Using TLS with RustPart I

time to read 4 min | 757 words

The next interesting step in my Rust network protocol exercise is to implement TLS. I haven’t looked at that yet, so it is going to be interesting.

The first search for “Rust TLS” gives me the rustls project, which seems to provide a native Rust implementation. There are also native-tls, which uses the system TLS library and binding for OpenSSL. I’m not sure how trust worthy rustls is, but I’m building a network protocol that is meant as an exercise, so I don’t really care. The first thing that I wanted to write was pretty trivial. Just convert the current code to use TLS on the wire, nothing more.

I wrote the following code to set things up:

There is a lot going on and I spend some time figuring out exactly what needs to be done here. I’m kinda proud / kinda scared of this code. It is packed. But on the other hand, if you read it, it is fairly clear, it is just that it does a lot of stuff.

I then used this in the following way:


And that led to this error when using OpenSSL to connect to it:


On the rust side, it died with:


The good thing, both sides agree that there is a handshake issue, but I had no idea what was going on. The bad thing, I have no clue why this is failing. According to the docs, this is supposed to work. I tried debugging into rustls and it seems to be failing very early in the handshake process, but I’m not familiar enough with either Rust or TLS to really understand why. The core issues seems to be here:


Note that we get a payload of None, but when I’m debugging through the code in the read_version method, it seems like it returns properly.

Oh, I figured it out, the issue is here:


Note the highlighted section? If this returns a None, it will silently return from the method. It is easy to miss something like that. Digging (much) deeper, I found:


And deeper still, we have:


Go back a bit and see what kind of URL I gave to OpenSSL, it was, and it seems like rustls doesn’t support raw IPs, only hostnames. The limit seems to be the result of this code:


This seems to be a known error, it is opened since May 2017 and it is a completely deal breaker for me. I’m guessing that this isn’t a big issue in general because usually if you have a cert, it is for a hostname and certificates for IPs (which exists, but tend to be issued for private CAs) are far rarer.

I changed my cmd line to use:


And it started working, but at that point, I was debugging this issue for over an hour, so that is quite enough for today.

As an aside, I was able to attach to the rust process using Visual Studio and debug it fairly normally. There is some weirdness that I believe relates to the cleanup, where if you abort a method early it looks like it goes back in time until it exit the method. However, overall it works fairly nicely. I do have to point out that many of the nicer features in the language are making debugging much harder.

That said, once thing was absolutely amazing and it was the fact that I was so easily debug into my dependencies, in fact, I was changing the code of my dependencies and re-running it, to help me narrow down exactly where the error was. That was easy, obvious and worked. Very nice!

More posts in "Using TLS with Rust" series:

  1. (17 Jan 2019) Authentication
  2. (11 Jan 2019) Part III–Will native tls do the trick?
  3. (07 Jan 2019) Part II - Client authentication
  4. (02 Jan 2019) Part I