Elevating Incident Response with the Ultimate Network Forensics

After a successful SOC @ RSAC2025, our team was stoked to be invited to help Cisco run the SOC at Cisco Live US (CLUS). We jumped at the chance to work with the Cisco team again. It was a great opportunity to innovate while helping Cisco protect and educate the attendees of the conference. Plus, the Cisco team is a lot of fun to hang out with—there’s a very infectious vibe in the SOC that has everyone buzzing for the entire week.

Packet capture is essential in the SOC. It provides an indelible record of all network activity, which is invaluable to the SOC team when investigating threats or security risks—hence the phrase, “PCAP or it didn’t happen.” For CLUS, we deployed two EndaceProbes with a combined storage of 864TB to continuously record all network activity delivered via 2 x 10GbE SPAN ports. This gave us the capacity to record at least several weeks of full network packet data—covering more than the entire duration of the show.

Endace Fusion integrations provided the glue between the Cisco Security suite and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk Enterprise Security/Splunk Cloud, Cisco XDR, Cisco Firepower, and Cisco Secure Network Analytics (SNA) through to EndaceVision and hosted Wireshark. When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk. Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the heavy use of insecure protocols, and ultimately driving automation to manage the unexpected volume. File-carving was enabled, and over 750,000 files were reconstructed from packet data, with over 40,000 samples submitted to Splunk Attack Analyzer (SAA) via Endace’s automatic submission software. SAA then sent over 12,000 files to Secure Malware Analytics (formerly Threat Grid) for dynamic analysis of the behavior.

SOC Findings and Lessons Learned

The SOC team was surprised and initially overwhelmed at the volume of unencrypted traffic on the network. Logging of passwords was coupled with a Cisco XDR automation that created an incident on each detection. This resulted in a heavy workload identifying and notifying users to educate and protect them in the future. The Splunk team developed a creative automated solution to notify users that the SOC detected their use of insecure protocols.

We even found a version of POP that was news to us all—APOP. This hashes the server timestamp in the response header with the user’s password to create a password digest. While this obscures the password, it only delays its inevitable retrieval, all the while the actual message bodies are still transferred in plain text!

In the theme of plain text passwords, reviewing the connections associated with one of these sessions showed a large number of file downloads in the Zeek log generated on EndaceProbe.  This was one of many clients that used the free conference Wi-Fi to download Windows update files, but after filtering out the cab files in a Splunk search, we found a suspicious-looking file:

A search on this filename in SAA confirmed the presence of a malware download by this unfortunate user, whom the SOC team made every effort to identify.

Building on our collaboration in the SOC at RSAC 2025, Endace was honoured to be invited by Cisco to provide EndaceProbe packet capture and a team of threat hunters to support the Cisco Live Security Operations Center (SOC). Our primary goal was to protect the network and attendees at Cisco Live San Diego 2025, our secondary goals were to educate and innovate.

Two EndaceProbes were deployed in Cisco’s SOC-in-a-Box, continuously recording all network traffic provided over two 10G SPAN ports.  We integrated with Splunk Enterprise Security and Cisco Security Cloud to deliver real-time network visibility. This enabled the SOC team to detect, locate, and address threats to attendees from external and internal sources, and helped to streamline and accelerate SOC investigation workflows.

The SOC at Cisco Live aims to protect the more than 23,000 attendees from threats on the conference network. When an attendee’s device is found to be compromised or unsecured, the SOC team makes every effort to identify, locate and help remediate the issue. The SOC team also provided public tours of the SOC to educate attendees about how the SOC is configured, the processes team members implemented, and the sorts of issues and threats that we uncovered.

Drawing on our experience at RSAC 2025 (where Endace captured 36 TB of data and 45 billion packets) we further optimized our approach for Cisco Live’s SOC.

The SOC was set up in two days, leveraging prior planning and our engineers’ experience at RSAC 2025.

At Cisco Live, Endace:

• Captured 78.9 TB of traffic representing 99.5 billion packets
• Reassembled more than 740,000 file objects in real time and sent 42,624 files to Splunk Attack Analyzer for further analysis
• Detected 2,256 occurrences of cleartext passwords being used from 92 unique devices
• Provided packet-level forensic evidence to all the SOC analysts for investigating and triaging a range of security events
• Integrated with Cisco XDR, Splunk, Cisco Firepower IDS, Splunk Attack Analyser, Cisco Secure Malware Analytics, Cisco Secure Network Analytics to provide a seamless IR workflow
• Innovated with new integrations that streamed additional metadata into Splunk, which sped up the investigation and remediation process

It was impressive to see the whole team of Cisco and Endace Engineers pivoting to the packet data to investigate an entire range of security incidents. Inspecting the packet data enabled fast decisions because the team could see exactly what was happening on the wire before, during, and after any event. As they say, PCAP or it didn’t happen!

Having complete packet data available at the SOC team’s fingertips really highlighted the incredible value of always-on packet capture; recording every packet and delivering complete, real-time visibility into threats, performance issues, and anomalous behaviour across the network.

There is an infectious energy that we get when working in a SOC environment like this, with Engineers from different teams and companies collaborating to track down threats, and share ideas on how to improve the SOC. The Endace team came away with many learnings that will improve EndaceProbe and our partner integrations.

We look forward to being a part of the SOC again at some more big upcoming events – stay tuned, details coming soon.