+972 52-548-6969
Production profiling security considerations
I am currently in the process of building support for the production profiling. The technical part (how to reduce the footprint of the profiler when not profiling) is pretty easy, and I spiked it out in a matter of a few hours. The non technical part has been driving me mad for days.
One of the major issues that I have to deal with with regards to production profiling is the issue of security. There are actually multiple levels to this issue. The most obvious one is that we want to secure who can start profiling. Initially I thought that providing a password and limiting IP range that can connect to the profiling endpoint would be enough, but I quickly realized that I am being overly naive.
The idea that someone can just hook into all the communication that goes on with the database is something that will give any security analyst heart palpitations. Not only do I need to secure access to the endpoint, I also need to ensure that no one can sniff this traffic, as it might very well include sensitive information. Beyond that, just to make sure that the aforementioned security analyst doesn’t show up at my door armed & pissed, it is important that only the system administrator (and not the developers) can turn that on.
That one isn’t so much to catch maliciousness, as much as to prevent people from deploying to production with their debug configuration enabled, but will also stop malicious acts of trying to open up holes in the application.
After some investigation, I decided that I probably want to use SSL (and the SslStream implementation in the BCL) to do that. This has several advantages, it means that the network traffic is encrypted and I can ignore that aspect. It also means that I can take advantage on mutual authentication to ensure that the client is really authorized to do connect to me. Finally, I can also require that the certificate will be installed on the server machine, thus meeting the requirement of the system administrator having to take an explicit step to allow it.
From design perspective, it looks nice, so the next question was to write some code to spike it. I should preface everything that follows with the admission that I have only rudimentary knowledge of the actual details of SSL and certificates. But basically, what I thought was to have the following:
- Server certificate – self signed, server auth, bound to a particular hostname, not CA.
- Client certificate – self signed, client auth, bound to a particular hostname, not CA.
The client certificate would need to be added as a trusted root on the server. That would allow mutual authentication between the two parties. The server will reject unknown certs or ones that do not bind to the same hostname as the server cert. I am not sure if I should require that the server cert would have to be installed on the client machine, but I am leaning on not requiring that.
Thoughts? Anything else that I might need to think about? Is this scheme really secure? Is it going to cause the ops people to go mad?
Comments
Another aspect of security is securing production data from developers. But I imagine a business with that level of concern would not allow profiling on production servers.
Chris,
That is why I wanted to have the client cert installed on the server, which is something that typically developers don't do.
IMHO using Server and Client certificates is overkill in this case because it assumes presence of additional security infrastructure and procedures such as Certificate Server and Certificate Issuing. It increases cost of system maintenance.
IMHO, Configuration attribute (to turn on profiling) + SSL + Domain authentication + IP restrictions is a well balanced architecture for this case.
TLS (SSL v3.1 is known as Transport Layer Security) is indeed the standard method of secure communication over a public network, providing encryption as well as one-sided or mutual authentication.
The canonical and slightly more complicated way to use PKI involves the sysadmin maintaining a CA certificate and a CRL (certificate revocation list) and using the CA cert to sign individual client certificates which he distributes to the devs. Signed client certificates are automatically approved unless the sysadmin revokes them by adding them to the CRL. This has the exact same effect as the solution you described, while being more canonical but more complicated.
Using the root certificate store to signal client trust seems hacky to me.
The purpose of the root certificate store is to store trusted CA-certificates, not client certificates. You seem to be trying to hack on a secondary semantic to this.
Consider what would happen if another subsystem used the same semantics for another purpose. You would not be able to authorize client-certificates for profiling without also authorizing it for the other subsystem.
You could use stunnel ( http://stunnel.org) to tunnel a plain TCP connection through SSL. It supports mutual client-server authentication with certificates and some other options also. This would save you lots of time & effort, I suppose.
Maksym,
SSL requires a certificate, and in order to allow the client to authenticate, it must be registered.
I am not sure you can do SSL without server cert, anyway
Justice,
The question is how complex it is going to be to maintain that for the average admin. And what will the dev with the parttime admin hat going to do?
rumburak,
Tunneling in .NET is a piece of cake, you only need SslStream
A lot of the time went into figuring out how to work with the certs, which is what would have happened anyway.
Ayende,
I meant that in case Profiler uses client certificates there is need to have Certificate Server installed otherwise customer should buy certificates from SSL Certificate Providers which not appropriate case for many companies.
Server certificate could be purchased from third party. As well as I know SSL connection could be established with server certificate only.
Maksym,
Oh, I am not talking about a Certificate Server, I am talking about a certificate that the server uses. Different things, similar names
Ayende,
I'm sorry for my English :)
It isn't your English, it IS confusing.
SSL using a server cert makes sense to secure the channel, but client certs are going to cause confusion. They're not used very commonly. Why not use Windows auth to assert that the user is in some particular group? You could even make that group configurable so the admin can choose to allow devs on an app-by-app basis or globally for all apps. The admin could use domain groups or local groups. Lots of options and well-understood by admins and devs alike.
I also don't like the idea of adding the client cert as a trusted root on the server side. With this, the client (or anyone that obtains its private key) will be able to issue certificates with any name that are accepted by the server.
You want to "trust" the client as a peer and not as an certification authority. There is a store for this purpose on windows, called "Trusted People" (on an EN install). WCF has a special validation mode that uses this store, however I didn't found a similar functionality for SslStream.
Another solution would be to use the RemoteCertificateValidationCallback (passed in the SslStream ctor) and a custom client certificate registry method.
I must admit I haven't really followed NProf too closely, but since you are using the SslStream class to build the communications channel, I assume that client's will be using the .NET Framework. In this case, why not use NegotiateStream instead and base your authentication mechanism on Kerberos (or NTLM)? This would provide for encryption and signing as well.
Furthermore, doing so would allow you to use the client's WindowsIdentity/*Principal to perform role-based access or impersonation. You could then rely on either AD or local accounts for authentication and alleviate the need to manage self-signed certificates.
Don't bother building in more security than Visual Studio remote debugger.
James,
Do not assume that I am running only on Windows, or that there is even a domain there to allow windows auth.
Joel,
I can assume neither a windows machine nor a domain existing there
Bruno,
VS remote debugger is generally not installed on production machines, and it is expected that it will be VERY invasive,
I do want to run this in production, and I want it to have as little effect as possible
I personally would use Symmetric encryption with a hash from a password... The profiler server is started without a password, any client want to listen need to have the same password.
I don't think asymmetric encryption is necessary in this case. Symmetric encryption is much faster and much simpler.
Firefly,
SSL doesn't do asymmetric encryption, it generates a symmetric key on the fly.
And this doesn't take care for the issues related to authentication and making sure that enabling that on production requires an explicit admin step
Certificates seem to be beyond KISS to me.
I'd go for a plain password or even some temporary invitation ticket.
Even assuming SSL is an easy thing to do in .Net, I wonder if the whole scenario is realistic. If there are perfrormance problems on a production system at customer site you will not limit yourself to watching NHibernate performance with NHProf but you will want to look at various system performance metrics, view application log files, check server configuration and do thousand other thingies. Shortly, you will want almost free access to the production machine and it doesn't matter if NHProf runs on SSL or not since you will be using secure communication channel (VPN?) anyway. I can't imagine a maintenance contract without access to the production machine.
rumburak,
There are many things that may require you to peek with the profiler, not all of them are full crisis mode, which is the only case where you have full & free access.
I want to enable no just that case.
In addition to that, you need to consider the issue of acceptance, you want to enable trust for that, otherwise it wouldn't get there in the first place.
Don't forget the overhead SSL produces. We recently had some problems after moving our web service to SSL.
We received a '413 Request Entity too large' error.
More info here: blogs.msdn.com/.../...-large-files-using-iis6.aspx
@Ayende,
SSL does both actually, it use asymmetric encryption to do the handshake then it use symmetric to transfer data but I think you know that already. So handing out the password is the same the server doing the handshake but not very convenience to support a large group of disconnected user (i.e. the internet) but for a small group like inside a company I think it work quite well.
Maybe I got the wrong perspective on the situation but I figure if it need to be that secure and the data are that sensitive then the number of client is probably small. After all it make no sense to set up all the security only to expose it to the whole company.
So I look at it from an administrator perspective. I would rather control a single password and hand them out as necessary than to set up a certificate server. Then again that's just me :) Your clients requirement might be different.
Firefly,
Yes, and since the handshake is only a very small percentage of the time we will use the connection, the cost of asymmetric encryption isn't really meaningful.
And where would you store that password? How do you make sure that only admin can manage that?
Firefly, you can also distribute one client certificate to multiple users if that is what you want.
Outside of network security, have you considered the ability to "redact" sensitive data from the trace stream? E.g. Flag sensitive columns like SSN so they don't show up in the trace output.
Kevin,
Sure, give me a general way to detect sensitive columns in arbitrary SQL.
In one of my systems the following gave you the credit card information:
select [2A871673-FBF4-4540-8F1D-89185A600119] from [2E113364-8E5D-40E5-8AE4-C7788C038488]
In some systems, email is considered sensitive, and there is a host of other stuff like that.
Trusting the developers to tell me about sensitive columns doesn't work either.
Obviously there's no way to deduce the sensitive columns automatically. I was assuming a configurable policy to identify the sensitive ones.
Kevin,
You CAN'T trust someone else to do the right thing when dealing with security
in that case look like you need to roll your own or hook into an existing authentication system. Since you want it to be simple and also cross platform it's probably easier to roll a simple role base authorization system with a simple GUI that the admin can add and remove users at ease. With role base not only you can make sure that only authorize users are monitoring but also know who is doing it. It's a little bit of work up front but I think the admin is also going to be happier :)
Comment preview