Category: Network visibility

By Caleb Millar, Staff Software Engineer, Endace

Summary

At Cisco Live APJC 2025 Endace provided Full Packet Capture and real-time File Reconstruction from network traffic as part of the Security Operations Center (SOC).

File reconstruction recreates files from network packet data and submits each reconstructed file to Splunk Attack Analyzer (SAA) and Secure Malware Analytics (SMA) to detect threats. The submit rate approaches 10,000 file samples for the busiest day at a large event, even after filtering.

To reduce load on SAA & SMA, new support was added to detect duplicate files across multiple Zeek VMs. We implemented this using a Splunk app key value store to act as a centralized database of previously submitted files. The result was a 70% reduction in the number of files submitted to SAA, and this allowed us to still record the blast radius of all devices that handled a potentially malicious file.

Optimizing Analysis of Reconstructed Packet Data

EndaceProbe provides continuous packet capture and simultaneously hosts virtual machines to analyze network traffic in real time using commercial or open-source tools.

At previous events, we deployed the capability to reconstruct files from packet data using Zeek. Every file extraction is logged with details such as IP address, port, and mime type. All Zeek logs are sent to Splunk for indexing and searching, and the file itself is sent to Splunk Attack Analyzer (SAA) for analysis.

Over a typical week of SOC operation, we would reconstruct and submit up to 30,000 files to SAA, so we set ourselves a task to reduce the number by not submitting duplicates of files that we already submitted.

SAA is an automated threat analysis tool that detects threats in files like phishing or malware using a series of threat analysis tools. For example, an `.exe` will be detonated in a sandbox environment to check for malicious behavior. SAA then assigns a score to each submission as an indicator of risk and may submit some files for further analysis to Secure Malware Analytics (SMA). The results and risk score are recorded in Splunk and Cisco XDR.

EndaceProbe also has built in capability for file extraction, accessible via the GUI, that allows a user to retrospectively analyze a set of captured packets to reconstruct files and generate Zeek logs. This is useful for reviewing any files that may not have been submitted in real-time.

For Cisco Live Melbourne, continuous packet capture was handled by two EndaceProbes, each running three Zeek virtual machines, for a total of six virtual machine instances.

Due to the high number of files extracted from captured packet data (at Cisco Live Melbourne we extracted 370,000+ files) it is not practical to upload every single file to Splunk Attack Analyzer. For this reason, we prioritize higher risk files (such as .exe) and skip uploading high volume/low risk files (such a .pem extracted from each TLS connection).

Since all file extractions continue to be logged and indexed in Splunk, it is always possible to re-run a file extraction using EndaceProbe by searching against relevant IP addresses. This supports the use-case where deeper investigation is required on a file which has not previously been uploaded to Splunk Attack Analyzer.

De-Duplicating Submissions

During the conference we noticed a high number of duplicate files being submitted to Splunk Attack Analyzer. It is not unexpected to find duplicate files, especially requests for software updates and other services accessed by many users. However, submitting duplicate files does create unnecessary resource cost with diminishing value. It also potentially delays submission and analysis of new and interesting files as the list of uploads are queued to distribute load over time.

• Therefore, we started working on a solution to skip previously submitted files. A de-duplication solution should have some key features:
• Support de-duplication across multiple virtual machine instances.
• Not introduce significant overhead if new API requests are required.
• Blast radius mapping – searching for all instances of a file that was seen on the network. Importantly, if a malicious file is detected, it should be possible to search and build events which check for all past and future instances.

To address these requirements, we settled on using a key-value store (KV store) as a centralized database for all files uploaded to Splunk Attack Analyzer.

One option for a KV store implementation is provided via Splunk apps. This solution had some key benefits:

• Allows virtual machines running Zeek to be re-deployed, or new instances added, while retaining the same duplication database. In practice, this means duplicates can be detected across all virtual machines running Zeek.
• Simplified deployment: since Splunk is already running as part of the SOC, there is no need to provision additional storage and other resources.
• Creates new opportunities to integrate key-value data with Splunk searches and apps.

The “key” for our de-duplication case is the SHA256 checksum of the file. SHA256 was used to match the checksum Splunk Attack Analyzer included on the submission result page.

Technically, this is enough to meet our requirements, but the additional value field in the key-value store allows additional metadata to be included for a given file submission. The file extraction logs were also updated to include the SHA256 checksum. This allows all instances of a file to be searched from within Splunk.

The Results

Enabling de-duplication provided immediate benefit. With many extracted files detected as duplicates, as seen in the figure. The items shown in purple are files which would have previously been submitted but now can be skipped.
In total, 6513 files were skipped due to duplication, with a total of 14369 files submitted to Splunk Attack Analyzer during Cisco Live APJC.

This shows immediate cost savings and creates an improved list of focused Splunk Attack Analyzer submissions. Integrating the duplication index with Splunk KV Store also provides additional future opportunities to integrate with other Splunk data and create new tools while keeping deployment simple.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Peter Watt, Senior Sales and Partner Integration Engineer, Endace

Summary

At Cisco Live APJC 2025 the SOC has a robust procedure in place to identify cleartext passwords, and through use of automation with Splunk/XDR to notify users directly via-email of their potential exposure – offering them support through the SOC on-site, during the conference.

This is a powerful capability of the SOC. It does however, require a valid email address – which is found within SMTP, IMAP and POP3.
When cleartext Passwords, and other PII are identified outside email based protocols … it becomes more difficult.

What we Identified

Firstly, through automation, the SOC was able to identify the use of a cleartext password:

From within the incident in XDR, a https reference link is provided to provide direct access to the packets of interest, stored on the Endace Packet Capture Appliance.

Using the session construction capabilities of Wireshark – we were able to reconstruct the data stream, and identify an FTP session had taken place with the following actions:

This exposed:

• External IP Address
• Username,
• Password
• Directory
• Filename
• Internal IP address
• Filetype being transferred

Missing:

• Email Address

Reverse lookup of the IP address was able to shed further light upon the situation. But we still needed additional information. Which is where the full packet capture became valuable.

Using Endace’s recorded full packet capture data, we extracted packet data between the source and destination IP’s and found that the FTP was also taking place on a non-standard port:

Upon reconstructing the session, the file being transferred was able to be reassembled too. The file format can be clearly seen in the following:

Piecing together all the components above, we were able to identify the source of the FTP, notify and educate them to resolve a potential security threat.

Having an FTP server open and accessible on the internet with clear text passwords is very risky indeed. An attacker that obtains credentials could easily access and download any private or sensitive content, and more concerning they could place infected binaries or malware on the server with the intention to infect and own any machines accessing content on that server. The users were shocked to learn about this exposure and immediately stopped using the FTP service.

The same methodology was used during threat hunting where other cleartext PII was identified.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Daniel Lawson, Senior Engineering Manager – Software, Endace
and Shaun Coulter, Technical Solutions Architect, Cisco

Summary

The Mission of the SOC at events like Cisco Live is:

Protect, Educate, Innovate.

While at Cisco Live APJC 2025, we were able to meet these objectives in a very direct way.

When Threat Hunting using full packet capture data we quickly identified a series of unencrypted Active Directory connections from conference attendees to their organization’s AD servers. Within an hour of identifying and documenting the issue the SOC had contacted the organization’s IT team and invited team members who were attending the conference to the SOC to walk them through the evidence.

We were only able to discover this serious security issue because EndaceProbe had recorded full continuous packet data for the entire conference, and EndaceVision provided us with a UI to rapidly threat hunt by searching, visualizing, and analyzing the packet data.

Curious Threat Hunt Leads to Rapid Discovery

During one of the SOC tours, one of my fellow engineers was discussing having identified some Kerberos traffic on the network. This prompted me to wonder if there was any LDAP traffic, as that is another common authentication protocol, and while it should be deployed with an encrypted communication channel, it isn’t always.

Sure enough, an EndaceVision query with an “Application is ldap” filter showed some unencrypted LDAP traffic!

Sending this traffic to Endace’s hosted Wireshark allowed me to view the raw packet decodes and confirm that these were completed connections, involving what looked like a service account and a password.

The device went on to request some entries from the organizations AD Global Address List, returning email addresses, job titles, and phone numbers. The GAL may also include other job and address information, and photos. Had an attacker found this information, at a minimum this password leakage could have resulted in the exfiltration of the organization’s complete address book, which may have been used for spamming or identify theft.

Depending on the nature of the service account that was leaked however, this might have resulted in a wider compromise: it might also have been a valid login account and therefore a foothold in the organization’s network, and potentially further scope for lateral movement within the network.

We used a Conversations Chords Chart to see if this involved other attendees, this showed us four local devices communicating with three LDAP servers for that organization.

The SOC was able to contact the organization involved and get in touch with the attendees, and they were invited down to the SOC where they got a detailed one-on-one demonstration of how we found the insecurity using the capabilities of Cisco and Endace SOC and the benefits of full packet capture.

The organization was both surprised and concerned with the data that had been exposed and stated that they would address the finding immediately.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/