By Caleb Millar, Staff Software Engineer, Endace
Summary
At Cisco Live APJC 2025 Endace provided Full Packet Capture and real-time File Reconstruction from network traffic as part of the Security Operations Center (SOC).
File reconstruction recreates files from network packet data and submits each reconstructed file to Splunk Attack Analyzer (SAA) and Secure Malware Analytics (SMA) to detect threats. The submit rate approaches 10,000 file samples for the busiest day at a large event, even after filtering.
To reduce load on SAA & SMA, new support was added to detect duplicate files across multiple Zeek VMs. We implemented this using a Splunk app key value store to act as a centralized database of previously submitted files. The result was a 70% reduction in the number of files submitted to SAA, and this allowed us to still record the blast radius of all devices that handled a potentially malicious file.
Optimizing Analysis of Reconstructed Packet Data
EndaceProbe provides continuous packet capture and simultaneously hosts virtual machines to analyze network traffic in real time using commercial or open-source tools.
At previous events, we deployed the capability to reconstruct files from packet data using Zeek. Every file extraction is logged with details such as IP address, port, and mime type. All Zeek logs are sent to Splunk for indexing and searching, and the file itself is sent to Splunk Attack Analyzer (SAA) for analysis.
Over a typical week of SOC operation, we would reconstruct and submit up to 30,000 files to SAA, so we set ourselves a task to reduce the number by not submitting duplicates of files that we already submitted.
SAA is an automated threat analysis tool that detects threats in files like phishing or malware using a series of threat analysis tools. For example, an `.exe` will be detonated in a sandbox environment to check for malicious behavior. SAA then assigns a score to each submission as an indicator of risk and may submit some files for further analysis to Secure Malware Analytics (SMA). The results and risk score are recorded in Splunk and Cisco XDR.
EndaceProbe also has built in capability for file extraction, accessible via the GUI, that allows a user to retrospectively analyze a set of captured packets to reconstruct files and generate Zeek logs. This is useful for reviewing any files that may not have been submitted in real-time.
For Cisco Live Melbourne, continuous packet capture was handled by two EndaceProbes, each running three Zeek virtual machines, for a total of six virtual machine instances.
Due to the high number of files extracted from captured packet data (at Cisco Live Melbourne we extracted 370,000+ files) it is not practical to upload every single file to Splunk Attack Analyzer. For this reason, we prioritize higher risk files (such as .exe) and skip uploading high volume/low risk files (such a .pem extracted from each TLS connection).
Since all file extractions continue to be logged and indexed in Splunk, it is always possible to re-run a file extraction using EndaceProbe by searching against relevant IP addresses. This supports the use-case where deeper investigation is required on a file which has not previously been uploaded to Splunk Attack Analyzer.
De-Duplicating Submissions
During the conference we noticed a high number of duplicate files being submitted to Splunk Attack Analyzer. It is not unexpected to find duplicate files, especially requests for software updates and other services accessed by many users. However, submitting duplicate files does create unnecessary resource cost with diminishing value. It also potentially delays submission and analysis of new and interesting files as the list of uploads are queued to distribute load over time.
- Therefore, we started working on a solution to skip previously submitted files. A de-duplication solution should have some key features:
- Support de-duplication across multiple virtual machine instances.
- Not introduce significant overhead if new API requests are required.
- Blast radius mapping – searching for all instances of a file that was seen on the network. Importantly, if a malicious file is detected, it should be possible to search and build events which check for all past and future instances.
To address these requirements, we settled on using a key-value store (KV store) as a centralized database for all files uploaded to Splunk Attack Analyzer.
One option for a KV store implementation is provided via Splunk apps. This solution had some key benefits:
- Allows virtual machines running Zeek to be re-deployed, or new instances added, while retaining the same duplication database. In practice, this means duplicates can be detected across all virtual machines running Zeek.
- Simplified deployment: since Splunk is already running as part of the SOC, there is no need to provision additional storage and other resources.
- Creates new opportunities to integrate key-value data with Splunk searches and apps.
The “key” for our de-duplication case is the SHA256 checksum of the file. SHA256 was used to match the checksum Splunk Attack Analyzer included on the submission result page.
Technically, this is enough to meet our requirements, but the additional value field in the key-value store allows additional metadata to be included for a given file submission. The file extraction logs were also updated to include the SHA256 checksum. This allows all instances of a file to be searched from within Splunk.
The Results
Enabling de-duplication provided immediate benefit. With many extracted files detected as duplicates, as seen in the figure. The items shown in purple are files which would have previously been submitted but now can be skipped.
In total, 6513 files were skipped due to duplication, with a total of 14369 files submitted to Splunk Attack Analyzer during Cisco Live APJC.
This shows immediate cost savings and creates an improved list of focused Splunk Attack Analyzer submissions. Integrating the duplication index with Splunk KV Store also provides additional future opportunities to integrate with other Splunk data and create new tools while keeping deployment simple.
Acknowledgements
Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.
The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.
Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc
For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/