Making Packet Forensics Easy

Original Entry by : Cary Wright

Extracting files and other information from recorded packet data

By Cary Wright, VP Product Management, Endace


Cary Wright, VP Product Management, EndaceRecorded network traffic often holds vital clues required to resolve serious Cyber Incidents, or difficult network or application issues. The challenge has been locating a packet guru with the skills to search and analyse recorded traffic to extract the vital evidence needed to resolve the issue at hand. Such skilful analysts can be a rare breed, so we have taken that expertise and packaged it into our latest EndaceProbe software.

Recorded network traffic is now faster to search from within existing security tools such as SIEM or SOAR, and extraction of files and other important information can be done by any team member with the click of a mouse.

Getting to the Packets Faster

Our integrations with partner solutions focus on making it quicker and easier for analysts to find and analyze the packet data they need to investigate and resolve incidents.

Analysts can go from an issue or alert in their security or performance monitoring tools directly to the related packet data in InvestigationManager™ with a click of the mouse. That can save hours of time extracting, downloading and carving-up massive .pcap files so they can be opened up in Wireshark®.

With EndaceVision, analysts can rapidly zoom the timeline in-and-out to look at pre-cursor or post event activity to understand the full scope of any event or alert. Analysis of packet data is done on EndaceProbe appliances at the place it was recorded using hosted Wireshark without having to download or transfer large .pcap files across your network.

Making packet data even more useful

In the past packet analysis has required deep expertise and experience with tools like Wireshark or Zeek used to extract essential information from the recorded packet data. This has made it difficult for less experienced analysts to extract value from packet data and often meant issues requiring packet forensics piled up on the desks of senior analysts to investigate.

With our latest software release (OSm 7.1), we’ve made it easy for even junior analysts to extract useful information from recorded packet data without requiring deep knowledge of packet structures and decode tools. Simply select traffic of interest in EndaceVision and with a single click extract malicious files, or generate detailed log data from all the selected packets. This makes investigating historical events fast, and far more efficient. And it does not require deep expertise – which means even junior analysts can perform packet forensics tasks.

Some examples of tasks that are made easier with the latest Endace software release include:

  • Reconstructing malware file downloads or transfers so you can submit them to a sandbox or virus tool.
  • Understanding exactly what data left your network by reconstructing file exfiltration events.
  • Easily generating logs from recorded traffic to look for things like unusual DNS activity, port scans, DDoS events, or other threatening activity.

See how easy this is in the short 10 minute demonstration below (file extraction is at 08:15):

For more information on these great new features, or to arrange a demonstration to show how Endace could help you, contact us.


Multi-Tenancy introduced with OSm 7.1

Original Entry by : Cary Wright

Securely sharing packet capture infrastructure across multiple entities

By Cary Wright, VP Product Management, Endace


Cary Wright, VP Product Management, EndaceWe are proud to announce that EndaceProbe now supports Multi-Tenancy, “Woo-hoo” I hear you say! If you are an MSPP, MDR, Service Provider, or organisation with multiple departments, your SoC teams can now reap the benefits of having access to weeks or months of continuously recorded network traffic whilst sharing costs with many other likeminded SoC teams. Let’s dig into what Multi-Tenancy is and why it’s important.

At the most basic level, Multi-Tenancy is the ability to host multiple “entities” (e.g. multiple customers or multiple organizational divisions) on a single architecture at the same time. To put it another way, Multi-Tenancy offers a way to share the costs of a system or service across more than one entity. Multi-tenancy can mean different things depending on your domain of expertise:

  • Cloud providers are inherently multi-tenanted, serving millions of clients with shared compute
  • Operating systems often host multiple tenants on a single machine
  • Networks can supply connectivity to multiple teams or organizations via a single infrastructure.

All these scenarios have these necessary requirements in common:

  1. Each tenant’s data must remain private and accessible to only that authorized tenant, and
  2. Each tenant needs access to reliable, predictable, or contracted resources – such as bandwidth, compute, storage, security services, expertise, etc.

Multi-tenancy can help organizations to scale critical security services in a cost-efficient manner. A capable security architecture/service requires a significant capability investment and the expertise to operate it. By enabling this investment to be shared, it enables services to be made available to organizations that might otherwise not have been able to afford them.

A good example of where Multi-Tenancy can be extremely useful is the Security Operations Center (SoC). Typically, only large, well-funded organisations have the resources to build their own dedicated SoC. Multi-tenancy can enable multiple organizations to share a SoC, each benefiting from a strengthened security posture without carrying the full burden of the costs and effort involved.

This is the model underpinning outsourced MSSP services, for example. But it can also be an ideal model for larger organizations with multiple divisions that each need to maintain separation from each other. Or where multiple individual companies are owned by a common parent. It can also be a useful way to safely isolate a newly acquired company until its systems can be safely migrated or transferred over to the new owner’s infrastructure.

We see lots of areas where organizations are benefiting from this ability to  share infrastructure and services. So we are very pleased to announce that with the new OSm 7.1 software release, EndaceProbe Analytics Platform now also supports Multi-Tenancy for network recording.

This is especially useful where multiple tenants share the same network. A single EndaceProbe, or a fabric of EndaceProbes, can now be securely shared across multiple different organisations or tenants, while keeping the data for each tenant secure and private. EndaceProbes continuously record all network data on the shared network, but only provide each tenant with access to their own data.

In this case the tenancies are defined by VLANs, where each tenant has a VLAN, or set of VLANs, that carries only their traffic. When a user needs to investigate a security threat in their tenancy, they simply log into InvestigationManager to search, inspect, and analyse only the traffic that belongs to that tenancy. It’s as if each tenant has its own, wholly separate, EndaceFabric, dedicated just to its own tenancy.

This new capability is important for large organisations that service multiple departments, agencies, or divisions. Service providers, MSPPs, and MDRs which service multiple clients will also benefit from Multi-Tenancy to give each of its clients ready access to its own recorded network traffic for fast, secure, and private, security incident response.

We are very excited that this new Multi-Tenancy feature can help make Network Recording accessible for many more organizations, helping them to resolve incidents faster and with greater confidence.

For more information on this great new feature, or to arrange a demonstration to show how Endace could help you, contact us.


Triggered vs Continuous Capture

Original Entry by : Cary Wright

Can security teams afford to collect only a fraction of the evidence necessary to run down cyber attacks?

By Cary Wright, VP Product Management, Endace


Cary Wright, VP Product Management, EndaceSecurity teams rely on a variety of data sources for investigating security threats, including logs from network security, endpoint detection logs, logs from various servers and AAA devices in the network. But often these logs are not sufficient to determine the seriousness and extent of the threat – they may be missing information or may have been manipulated by an attacker. So many analysts turn to continuous Full Packet Capture (FPCAP) to fully analyse threat activity.

Continuous Full Packet Capture records every packet (regardless of rules) to disk in a large rotating buffer, ensuring that any and all threat traffic is recorded for later analysis. FPCAP will reveal all the various stages of an attack, including the initial exploit, command and control communications, surveillance and execute phases. FPCAP is very trustworthy because it cannot be manipulated by an attacker. And because all packets are being recorded we can be sure we have a record of all threat activity, whether that threat is detectable by security monitoring tools or not. Historically, it has been expensive to deploy enough FPCAP storage to get the recommended 30 days of look-back needed by security analysts. This is where triggered capture enters the story.

What is Triggered Packet Capture?

The concept of triggered packet capture is to capture only those packets related to detected or likely threats and discard the rest. The goal is to reduce the storage capacity needed to provide the necessary look-back period and thereby minimize the cost of storage.

One approach is to create rules or triggers to capture specific sequences or flows of packets and ignore anything that doesn’t match these rules. Recorded packets are stored in a unique packet capture (PCAP) file on a local file system or RAID array and at the same time an event notification may be logged to a SIEM or other security monitoring tools to ensure that PCAP can be found later.

Alternatively, the trigger rule might be a firewall signature or rule that causes a handful of packets to be recorded for that event to show what triggered the rule to fire. This is often an option provided by NGFWs but typically only very limited storage capacity is provided.

A second approach is to only capture those packets that your security monitoring tools don’t recognise as “normal” on the basis that this anomalous traffic could potentially indicate a threat or attack.

The Problems with Triggered Packet Capture

There are serious flaws with both the approaches above that will leave security teams lacking the PCAP evidence they need to properly investigate events.

Firstly, it’s really not possible to predict ahead of time what attacks you might be subject to. What about attacks that leverage the next Heartbleed SolarWinds or Log4J 2 type of vulnerability. How can you write triggers or rules that will ensure you reliably capture all the relevant packets related to future unknown threat vectors and vulnerabilities?  More than likely you will miss these attacks.

Secondly, capturing the packets relating to an initial trigger event such as an intrusion is useful, but it only tells part of the story. An initial exploit is just the beginning of an attack. You also need to capture downstream activity such as command and control communications, lateral movement, surveillance, exfiltration and other malicious activity. These will often be unique to each victim once the initial exploit has been successfully executed. And to investigate these conclusively you need access to full packet data. You can’t reconstruct exfiltrated data, or malware drops from metadata.

Lastly, attackers often try to camouflage their activity by using common protocols and traffic patterns to make their malicious activity look like normal user behavior. Capturing only “unknown” or “anomalous” traffic will most likely miss this activity.

These problems with triggered capture become very costly because your team have only part of the story, they must start guessing what might have happened and they must assume the worst case when reporting to customers, markets or affected parties. The lack of concrete evidence becomes very costly, just to save a few dollars on storage.

Only Full Packet Data provides a Complete Picture

During the incident response process, once an exploit has been detected analysts then need to look at any subsequent activity from the affected host or hosts – such as payload downloads, command-and-control (C2) communications, lateral movement to other hosts on the network, etc.

Only by examining full packet data can we really start to understand the full impact of a detected threat. If we only have triggered capture available and we don’t have any packets captured relating to activity after the initial exploit was detected, how can we tell whether there was a backdoor installed? Or tell what data was exfiltrated so we can understand what our legal obligations are to notify affected parties. These sort of questions cannot remain unanswered.

Advances in storage media, capture techniques and compression technology have made continuous full PCAP affordable now. So there’s no longer a need to compromise by deploying unreliable Triggered Capture to save a few bucks on disk storage. It’s simpler, faster, and much more robust to capture every packet so we are prepared to investigate any threat, new or old. Continuous full PCAP ensures teams can respond to incidents quickly and confidently, and in the long run this significantly reduces the cost of cybersecurity.


Wireshark without the wait!

Original Entry by : Cary Wright

With Wireshark on EndaceProbe you can quickly search hundreds of Terabytes of packet data to analyze important packets in Wireshark

By Cary Wright, VP Product Management, Endace


Cary Wright, VP Product Management, Endace

Who can afford to wait when responding to a critical security incident? With Wireshark now hosted on EndaceProbe we have eliminated all the waiting around to see packet evidence. Reviewing captured network history will often reveal vital evidence needed to remediate a threat, evidence that may have been wiped from system logs.

Unfortunately, if you’re using Wireshark on your desktop to view that evidence you know it can be a very slow process. Just downloading a multi-GB capture file from your capture appliance can take a while, and then loading it up on your desktop can also be a lengthy process.  All this waiting and context switching is a productivity hit for you and your team– not to mention a data privacy risk if those PCAPs are sitting on your desktop or laptop.

I’m excited to say there will be no more waiting around to view packets with our newly released OSm 7.0 software! A full instance of native Wireshark is now hosted right on each EndaceProbe appliance so you can review captured network traffic quickly and securely. We have also included WireShark on each Endace InvestigationManager instance, allowing you to search over up to 100 EndaceProbes in parallel and present a single merged packet view inside Wireshark.

There is no need to download large PCAPs over the network, and no need to store them insecurely on your desktop PC or laptop to view in Wireshark. Viewing network packet captures is now lightning fast because EndaceProbe high-performance hardware serves the packets from the local RAID directly to a Wireshark instance hosted on the EndaceProbe.

If you’re a regular Wireshark user you will know that Wireshark doesn’t handle large PCAPs very well, just loading a 1GB file can take forever let alone a 100TB pcap. With Wireshark on EndaceProbe you can now quickly search hundreds of Terabytes of packet data to view or analyze important packets in Wireshark. The workflow is much faster and more secure. And Wireshark power users will be glad to know it’s a full Wireshark instance with all the useful features and decodes that you’ve come to know and love.

Here’s a sneak preview:

Wireshark on EndaceProbe with OSm7
With OSm 7.0, now you can go directly from EndaceVision to Wireshark hosted on EndaceProbes – without having to download large pcap trace files.

Endace + XSOAR = Nirvana for the SoC

Original Entry by : Cary Wright

Integrating Palo Alto Cortex XSOAR with the EndaceProbe Analytics Platform

By Cary Wright, VP Product Management, Endace


Cary Wright, VP Product Management, EndaceThis week we are announcing an exciting integration with Palo Alto Networks Cortex XSOAR, formerly Demisto. This integration provides XSOAR customers with automated playbooks that easily pull in packet-level evidence for fast, conclusive, and repeatable response to security incidents. This integration complements our existing partnership with Palo Alto Networks NGFW and Panorama so now you can access packet-level data across multiple Palo Alto solutions.

So what is this “Nirvana for the SoC” we are all striving for?

The most effective SoC teams I’ve seen are well-oiled machines, reviewing and resolving many potentially dangerous security incidents each day and neutralizing threats quickly and confidently. What makes these teams successful is a repeatable and well-understood process, based on evidence, backed by automation, with integrated workflows across a suite of best in class security tools.

These teams have a wide range of experience–from new recruits to seasoned experts–all highly motivated and working collaboratively to solve complex issues. This exceptional environment not only provides high levels of productivity and security, but it also is great for team morale, staff retention, and hiring. Adding new staff is streamlined because all the processes are documented and/or automated, workflows are simple, and less experienced hires can contribute quickly. I am sure you would agree this is the SoC team Nirvana that we are all striving for?

SoC teams are flying blind without network packet history at their fingertips. Sophisticated attackers do their best to cover their tracks by modifying server logs or deleting evidence. However,   packets don’t lie and can’t be tampered with. That’s why many SoC teams deploy EndaceProbe alongside their firewalls so they can turn to the packets to investigate their most challenging security incidents. It’s the evidence needed to know without a doubt what happened at 2pm last Tuesday afternoon when a security alert indicated a potential attack.

We integrated with Cortex XSOAR because we realized that many teams were missing the essential packet-level evidence required for fast and conclusive security investigations. XSOAR playbooks now automate the collection of packet evidence from any EndaceProbe in the deployment. Packet evidence is then archived and attached to a “case” or “war room” allowing multiple team members to contribute to the investigation at any time in the future.

The complete workflow can be integrated with the entire security tool suite including endpoint, network, SIEM, NGFW, and other security elements. And finally, these playbooks can be customized to suit the specific needs of the organization.

Check out the demo video on Palo Alto Network’s Fusion partner page to see this integration in action, and reach out if you’d like more information.

I am very proud of what our team has achieved with this integration to Cortex XSOAR. Our customers can now manage alerts across all sources using a standard process, take action on threat intel, and automate response for any security use case – resulting in significantly faster responses that require less manual review. I’m really looking forward to seeing our customers take advantage of this new capability to create their own SoC team Nirvana.

Happy hunting,

Cary