Author: Cary Wright

Can security teams afford to collect only a fraction of the evidence necessary to run down cyber attacks?

By Cary Wright, VP Product Management, Endace

Security teams rely on a variety of data sources for investigating security threats, including logs from network security, endpoint detection logs, logs from various servers and AAA devices in the network. But often these logs are not sufficient to determine the seriousness and extent of the threat – they may be missing information or may have been manipulated by an attacker. So many analysts turn to continuous Full Packet Capture (FPCAP) to fully analyse threat activity.

Continuous Full Packet Capture records every packet (regardless of rules) to disk in a large rotating buffer, ensuring that any and all threat traffic is recorded for later analysis. FPCAP will reveal all the various stages of an attack, including the initial exploit, command and control communications, surveillance and execute phases. FPCAP is very trustworthy because it cannot be manipulated by an attacker. And because all packets are being recorded we can be sure we have a record of all threat activity, whether that threat is detectable by security monitoring tools or not. Historically, it has been expensive to deploy enough FPCAP storage to get the recommended 30 days of look-back needed by security analysts. This is where triggered capture enters the story.

What is Triggered Packet Capture?

The concept of triggered packet capture is to capture only those packets related to detected or likely threats and discard the rest. The goal is to reduce the storage capacity needed to provide the necessary look-back period and thereby minimize the cost of storage.

One approach is to create rules or triggers to capture specific sequences or flows of packets and ignore anything that doesn’t match these rules. Recorded packets are stored in a unique packet capture (PCAP) file on a local file system or RAID array and at the same time an event notification may be logged to a SIEM or other security monitoring tools to ensure that PCAP can be found later.

Alternatively, the trigger rule might be a firewall signature or rule that causes a handful of packets to be recorded for that event to show what triggered the rule to fire. This is often an option provided by NGFWs but typically only very limited storage capacity is provided.

A second approach is to only capture those packets that your security monitoring tools don’t recognise as “normal” on the basis that this anomalous traffic could potentially indicate a threat or attack.

The Problems with Triggered Packet Capture

There are serious flaws with both the approaches above that will leave security teams lacking the PCAP evidence they need to properly investigate events.

Firstly, it’s really not possible to predict ahead of time what attacks you might be subject to. What about attacks that leverage the next Heartbleed SolarWinds or Log4J 2 type of vulnerability. How can you write triggers or rules that will ensure you reliably capture all the relevant packets related to future unknown threat vectors and vulnerabilities?  More than likely you will miss these attacks.

Secondly, capturing the packets relating to an initial trigger event such as an intrusion is useful, but it only tells part of the story. An initial exploit is just the beginning of an attack. You also need to capture downstream activity such as command and control communications, lateral movement, surveillance, exfiltration and other malicious activity. These will often be unique to each victim once the initial exploit has been successfully executed. And to investigate these conclusively you need access to full packet data. You can’t reconstruct exfiltrated data, or malware drops from metadata.

Lastly, attackers often try to camouflage their activity by using common protocols and traffic patterns to make their malicious activity look like normal user behavior. Capturing only “unknown” or “anomalous” traffic will most likely miss this activity.

These problems with triggered capture become very costly because your team have only part of the story, they must start guessing what might have happened and they must assume the worst case when reporting to customers, markets or affected parties. The lack of concrete evidence becomes very costly, just to save a few dollars on storage.

Only Full Packet Data provides a Complete Picture

During the incident response process, once an exploit has been detected analysts then need to look at any subsequent activity from the affected host or hosts – such as payload downloads, command-and-control (C2) communications, lateral movement to other hosts on the network, etc.

Only by examining full packet data can we really start to understand the full impact of a detected threat. If we only have triggered capture available and we don’t have any packets captured relating to activity after the initial exploit was detected, how can we tell whether there was a backdoor installed? Or tell what data was exfiltrated so we can understand what our legal obligations are to notify affected parties. These sort of questions cannot remain unanswered.

Advances in storage media, capture techniques and compression technology have made continuous full PCAP affordable now. So there’s no longer a need to compromise by deploying unreliable Triggered Capture to save a few bucks on disk storage. It’s simpler, faster, and much more robust to capture every packet so we are prepared to investigate any threat, new or old. Continuous full PCAP ensures teams can respond to incidents quickly and confidently, and in the long run this significantly reduces the cost of cybersecurity.