By Jeff Brown, director of training, U.S. global support, Endace division of Emulex
On April 7, the “Heartbleed” bug was announced. It’s a serious flaw in the OpenSSL 1.0 – 1.0.1 code series which affects all applications using it for encryption. In short, it means that anyone who can connect to the server can remotely read the server’s memory – including the SSL certificate secret key, usernames and passwords, and anything else.
With the Heartbleed bug exploit code in the wild, anyone can take advantage of the critical time between public exposure of the exploit and when all organizations can patch (or take offline) vulnerable systems. So, for almost every organization in the world, there are three questions that come to mind. The first question is “which of my public facing servers is vulnerable?” The second question is “have I been exploited since this became public?” And the third question is “what have I lost?”
The EndaceProbe™ Intelligent Network Recorder (INR) helps answer all three questions.
Which of my public facing servers is vulnerable?
The first step is to use your database (you DO have a database matching services, servers, and operating systems, right?) to locate those systems known to be vulnerable and that are public facing. Take them offline and patch them. Those are the knowns. Now, what about the unknowns?
You cannot use the presence of malformed heartbeat requests to confirm or deny vulnerability – that just tells you somebody is attacking, which is perhaps a common event these last few days! It is the heartbeat response that identifies whether a server is vulnerable. So what you need is to send each of your servers an exploit request and then filter on just heartbeat responses from vulnerable servers. As it turns out, that is surprisingly easy if you have an EndaceProbe INR monitoring your network.
First, download the exploit code off the Internet, set it up on a workstation running outside your firewall on a known IP address X. Have it run the exploit against every IP address in your domain. That’s what the bad guys are going to do … beat them to it! If you can’t set up your own attack system, there are websites already online that will attack for you. You just need to send them your IP addresses to attack.
Next, on the EndaceProbe INR which is monitoring traffic just inside your firewall (you do have an EndaceProbe INR capturing your firewall traffic, don’t you?), set up a visualization that is filtering bi-directionally on IP address X – your attack workstation’s address. That will isolate the exploit attempts and responses. This filtering will result in a small amount of data over the length of time it takes for your exploit workstation to work through your IP address space.
From this visualization, click on the “packets” view and enter the following display filter:
(ssl.record.content_type == 24) && (ssl.record.length > 64)
The (ssl.record.content_type == 24) identifies all heartbeat requests and responses. Heartbeat requests (both valid requests and exploit requests) are typically less than 64 bytes long. Valid heartbeat responses should also be less than 64 bytes. So the (ssl.record.length > 64) should only catch responses returning lots of data back to your attacking workstation. That means every packet that matches the above display filter is probably from a server that is vulnerable. Locate the server by its IP address, pull it offline and patch it.
Have I been exploited?
Until April 7, this bug had been undiscovered (publicly), but it has existed in versions of the OpenSSL code for more than two years. It is therefore very difficult for an organization to fully determine its overall risk of having been exploited if someone discovered the bug earlier and has been using it nefariously. But what we do know is that the bad guys are most certainly monitoring vulnerability releases, especially ones that are accompanied by simple-to-use exploit code! Therefore, it stands to reason that an organization’s risk of exploit is highest between public disclosure of the exploit and time-of-patch.
So having an EndaceProbe INR with even a few days’ worth of storage allows the organization to perform an exhaustive post-mortem for those critical hours or days of maximum risk. Fortunately that EndaceProbe INR you have sitting behind your firewall will have captured 100 percent of the traffic from the last few days. Time to put it to use!
From step one above, you now (hopefully) have a short list of IP addresses for servers that are vulnerable. To make the search efficient, first look for the exploit attempt, and then for the response. This two-step process works best because:
- The amount of traffic into the server is typically much less than out. It is faster to search the traffic coming in.
- The exploit arrives on port 443, so is easy to filter on that port. The response can go out on any port number.
It it is therefore much faster to find the exploit than to find the response, so only look for the response, if you know the exploit has occurred.
Going through your vulnerable IP addresses one at a time, use a visualization that filters on the server’s destination IP address and destination port 443. (If you use other ports for SSL you’ll want to check that traffic too.) Now launch Endace Packets™ and enter:
((ssl.heartbeat_message.type == 1) && (ssl.heartbeat_message.payload_length > 61))
This filter might result in some false positives depending on whether or not there are legitimate clients out there that use heartbeat payloads > 61 bytes, but 61 seems to be the common number used. This filter will identify heartbeat request packets where the ssl.heartbeat_message.payload_length is larger than normal – a strong indication of an exploit attempt.
If you see any results from this filter, then it is time to look at the heartbeat response. So, back to your visualization! Filter on the attacker’s IP address as the destination. You could just stop there and look at everything sent to the attacker on any port, but depending on how much traffic that is, you might want to step through one vulnerable server at a time. If slow and steady is your style, then you will also filter on the source IP address of the vulnerable server detected above, with destination port taken from the heartbeat request packet.
Now, launch Endace Packets and enter the same exploit response filter you used before:
(ssl.record.content_type == 24) && (ssl.record.length > 64)
This will identify if the server responded to the exploit. You’ve already confirmed that server is vulnerable, so it probably sent a large amount of RAM data back to the attacker. Bad news, but at least you know for sure you’ve been exploited. Now…
What have I lost?
The heartbeat response will consist of several IP packets forming a single TCP PDU. Overall size of the PDU will depend on how large the (false) payload size was in the exploit heartbeat request. The response PDU is easy to identify in Endace Packets, but it is encrypted so you won’t be able to see what is inside. Time for Wireshark!
Use the EndaceProbe INR download capability to download the exploit session to your workstation. You’ll need to get the private SSL key from the exploited server, load it in Wireshark, decrypt the response message, and determine whether anything important is there. It’s time-consuming work, but well worth knowing what, if anything, has been lost!
What about workstations?
The SSL heartbeat is symmetrical, so, in theory, an OpenSSL client can be attacked by a malicious server just as easily as a server can be attacked by a client. This should be your next concern. Windows and Mac appear to be safe, but what about your Linux workstations? Workstations are harder to test because they won’t respond to a direct attack. They have to go to a malicious website before you will see any exploit heartbeat requests coming to them.
Good luck with your mitigation efforts, and please let us know if there’s anything we can do to help with this process.