Endace Packet Forensics Files: Episode #63

Original Entry by : Michael Morris

In Episode 62, Michael talks to Jack Chan, Vice President, Product Management at Fortinet

By Michael Morris, Director of Global Business Development, Endace

Michael Morris, Director of Global Business Development, Endace

Why NDR is Evolving—And What Enterprises Should Demand From It

In this episode of The Packet Forensic Files, I spoke with Jack Chan, VP of Product and Field CTO at Fortinet, about what sets a strong Network Detection and Response (NDR) solution apart. Jack explained that while many vendors claim to offer NDR, the best solutions help security teams see deep into their networks, spot threats early, and look back in time to understand what really happened before and after an incident.

Jack pointed out that most companies already have too many security tools creating alerts. That’s why it’s so important for NDR to work well with other tools like EDR—so SOC teams know which alerts really matter. He also shared how AI and machine learning are helping to detect threats even in encrypted traffic, and how newer tech like generative AI is making it easier for analysts to investigate issues without writing complex queries.

We talked about the benefits of using NDR alongside firewalls. Since NDR is passive, it can show you how clean or risky your network is without disrupting anything. But when NDR spots a threat, teams need to decide—should it trigger an automatic response or wait for human approval? Jack recommends using automation carefully, with some human oversight.

Finally, Jack reminded us that technology alone isn’t enough. Security starts with people—whether it’s developers writing secure code or staff avoiding risky clicks. No matter how advanced your tools are, the human factor still plays a huge role in keeping networks safe.

Don’t miss this episode as Jack shares practical tips, real-world examples, and a clear-eyed view of where the NDR space is heading.

 

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.

Threat actors are recording PCAPs, maybe you should too?

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace

Barry

On Aug 29th, Government Cybersecurity agencies from around the world released a joint advisory detailing how nation-state threat actors are compromising networks across the world, particularly in the US, Australia, Canada, New Zealand and the UK: www.cisa.gov/news-events/cybersecurity-advisories/aa25-239a www.darkreading.com/cybersecurity-operations/cisa-fbi-nsa-warn-chinese-global-espionage-system

These attacks are primarily focussed on telecommunications, government, lodging and military networks, and the tactics, techniques, and procedures (TTP) overlaps with APT (Advanced Persistent Threat) actors linked to multiple China-based entities. These threat actors are exploiting well-known vulnerabilities in VPN servers and web user interfaces on switches and routers.  Even devices not owned by targets of interest are being compromised in order to provide additional attack pathways to the intended targets.  Upon gaining a foothold in a network, persistence is achieved by modifying ACLs, opening services on non-standard ports to avoid detection, and tunnelling C2 and exfiltrated data to obfuscate malicious activity.

Of note in this particular instance is the use of PCAP collection on the target network by the threat actors.  Once they’ve gained a foothold on the network infrastructure, the native capability of some routers to record PCAPs is then used to capture TACACS+ (authentication) traffic.  When transmitted in clear text (or weakly encrypted) this authentication traffic exposes users credentials which can then be used to elevate the attacker’s access and enable them to move laterally across the network.

The use of network sniffing to extract credentials in authentication traffic is a common technique of threat actors (attack.mitre.org/versions/v17/techniques/T1040/).  As we continue to see the stubborn use of unencrypted and weakly-encrypted protocols on networks, these insecure communications remain prime targets for credential gathering.  Additionally, the uses of maliciously collected PCAP is evolving, with the ArcaneDoor campaign taking this a step further and exfiltrating captured PCAPs for remote analysis (attack.mitre.org/campaigns/C0046/).  Exfiltrated PCAPs may contain anything from authentication data to file objects.

PCAP data is the ground-truth for what is happening on the network, and it is the source that all other network and security telemetry is derived from.  Threat actors know this and value the raw unfiltered and unsampled intel that it provides about the target.  This begs the question: if your adversaries see value in collecting PCAPs off your network, shouldn’t you be capturing full PCAP too!? 

If you are not recording your network traffic, your security team has less visibility into network activity than your attackers – which makes the job of protecting your network impossibly difficult. With PCAP at their fingertips, SOC analysts can see exactly what’s happening on the network, making for faster, more accurate investigation and resolution of security incidents – as this excellent blog post from Cisco’s Steve Nowell describes.

That full PCAP data is so valuable to attackers also highlights a stark warning that wise defenders should heed. PCAP data must be protected and secured to the highest standards.  Can you trust packet capture solutions that aren’t FIPS and Common Criteria certified? Or packet capture sources that can’t be properly locked down and protected from access by attackers?

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/ 

Network Visibility in Action: Endace and Cisco Drive SOC Defenses at RSAC 2025

Original Entry by : Cary Wright

By Cary Wright, VP Product Management, Endace

Cary Wright, VP Product Management, Endace

Uncovering insights from the 6th Annual Security Operations Center at RSA Conference

For the sixth consecutive year, a dedicated Security Operations Center (SOC) monitored the RSA Conference (RSAC) network, protecting a dynamic environment serving over 40,000 attendees.  A collaboration between Endace and Cisco (and other security partners), the SOC provided real-world insights into current threat landscapes and security challenges, and demonstrated the critical importance of comprehensive network monitoring and real-time threat detection in large-scale environments.

The 2025 SOC team consisted of:

  • 5 Endace analysts
  • 9 Cisco/Splunk analysts
  • 3 dedicated threat hunters
  • 3 managers

 

Network Monitoring at Unprecedented Scale

The SOC captured and analyzed an astounding volume of data flowing through the conference network:

  • 40+ billion packets captured (more than double the 19 billion from the previous year)
  • 33 TB of packet data (up from 17TB)
  • Peak bandwidth usage of 3.4 Gbps (up from 2.2 Gbps)
  • 615 million total sessions (increased from 383 million)
  • 793 million logs captured
  • 287,000 files extracted with 26,374 submitted for deeper analysis

Endace’s VP of Product Management Cary Wright explained the scope: “We tapped into the network and recorded everything—all the packets that traveled across that network—approximately 30 terabytes of data over the course of the whole conference.”

The Technical Architecture: Integration in Action

The SOC implemented a sophisticated, multi-layered security architecture centered around visibility and integration:

    1. Network Capture Layer: EndaceProbe appliances performed full packet capture, creating a complete record of all network activity.
    2. Log Generation and Analysis: The Endace systems generated metadata through tools like Zeek, which was then forwarded to Splunk and Cisco security tools for analysis.
    3. Threat Detection Systems: Cisco Secure Firewall provided intrusion detection (running in non-blocking mode to avoid disrupting vendor demonstrations while still identifying potential threats).
    4. Integration Layer: All components were interconnected, allowing analysts to pivot seamlessly from alerts directly to the relevant packet data, providing context for rapid investigation.
    5. File Analysis Pipeline: Files transmitted across the network were extracted and analyzed: 
      • 287,000+ files extracted from network traffic
      • 26,374 files sent to Splunk Attack Analyzer
      • 7,546 files forwarded to Cisco Malware Analytics for in-depth examination

Key Security Findings and Trends

The SOC’s monitoring revealed several concerning security trends:

1. Declining Encryption Levels

One surprising finding was a drop in the percentage of encrypted traffic, from approximately 80% in 2024 to 74% in 2025. This regression toward “the dark past” of unencrypted communications creates significant security vulnerabilities.

More troubling was the increase in weak encryption (TLS 1.0/1.1) to 40% of encrypted traffic, along with the continued presence of plaintext password transmission.

2. Plaintext Passwords Continue

Though trending downward over the years, plaintext passwords remain a persistent problem, showing that the power of a strong password is nothing without an encrypted communication protocol!

      • 2020: 96,361 cleartext passwords (2,178 unique accounts)
      • 2022: 55,525 cleartext passwords (2,210 unique accounts)
      • 2023: 36,910 cleartext passwords (424 unique accounts)
      • 2024: 20,916 cleartext passwords (99 unique accounts)
      • 2025: 1,807 cleartext passwords (87 unique accounts)
3. Legacy Protocol Persistence: POP3 Refuses to Die

The SOC discovered continued use of vulnerable legacy protocols:

      • POP3 (unencrypted email retrieval)
      • Non-secured SMTP (email transmission)
      • Unencrypted IMAP
4. Advanced Threat Techniques

The SOC identified several sophisticated attack techniques, including:

      • New domain generation algorithm (DGA) approaches using combinations of 2-3 random words
      • Command and control (C2) traffic
      • Cleartext transmission of sensitive data
      • Unsecured translation services transmitting text and audio in the clear
      • Exposed CCTV camera feeds

The Value of Complete Network Visibility

The collaborative SOC deployment at RSAC 2025 demonstrated the crucial role that full packet capture plays in modern security operations. By capturing and analyzing every packet traversing the network, security teams gained:

      • Complete visibility into all network communications
      • Contextual evidence for security investigations
      • Rapid response capabilities through integrated tools
      • Retrospective analysis of historical network data

The integration between Endace’s packet capture technology and Cisco’s security suite enabled a powerful workflow: alerts from security tools could be immediately investigated by pivoting directly to the relevant network traffic, dramatically reducing investigation time.

Key Takeaways for Security Teams

Based on the RSAC 2025 SOC experience, organizations should consider these best practices:

      • Deploy comprehensive network monitoring with full packet capture for complete visibility
      • Implement integrated security tools that work together seamlessly
      • Focus on encryption enforcement to protect sensitive data in transit
      • Eliminate legacy protocols that transmit data in cleartext
      • Use personal VPNs when connecting to public networks
      • Keep operating systems patched and maintain robust configuration management

The Endace and Cisco-powered SOC at RSAC 2025 demonstrated that comprehensive network visibility remains fundamental to effective security operations. As threats grow more sophisticated, the ability to see, analyze, and respond to every packet traversing the network becomes increasingly critical.

By integrating full packet capture with advanced security analytics, organizations can build security operations centers that provide both the breadth and depth of visibility needed to detect and respond to today’s most sophisticated threats.

This blog post is based on information shared during the “PROTECTED: The 6th Annual Report from the SOC at RSAC” session at RSA Conference 2025.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/ 

Accelerating Security Operations at Black Hat USA 2025

Original Entry by : Anantha Srinivasan

Fast Queries and automated PCAP Workflows

By Anantha Srinivasan, Senior Software Manager, Endace and Sundarram Paravastu, Principal Software Engineer, Endace

In the fast-paced environment of Security Operations Centers (SOCs), swift incident response and threat hunting are essential to combat sophisticated cyber threats. Slow queries, siloed workflows, and manual PCAP retrieval significantly hinder SOC operations, delaying investigations and prolonging detection and remediation times. These inefficiencies, compounded by limited integration and automation, exacerbate analyst fatigue and increase the risk of overlooking critical threats.

At the Black Hat US 2025 NOC, several key innovations were developed and enhanced with Endace and the integrations we have with Cisco, Palo Alto Networks and Corelight. Read: Endace Always-on Packet Capture for additional details.

The innovations developed at Black Hat facilitated the use of Endace’s always-on packet data in incident response and threat hunting workflows, empowering analysts to leverage deep forensic insights more quickly and intuitively during investigations.

Meta-data, flow data or partial packet capture can be useful in detecting possible threats and enabling analysts to get an overview of threat activity. But it often doesn’t contain enough detail for analysts to be certain about exactly what happened.

Always-on packet capture ensures analysts have a complete record of all network activity, giving them access to definitive evidence for threat investigations. Not only can they investigate specific events, but they can also look at what else happened before, after or during the specific event they are looking into.

This lets analysts build a complete picture of threat activity that is otherwise difficult or impossible without access to complete packet evidence. With detailed forensic evidence at their fingertips, they can make better decisions more quickly, speeding investigations and enabling more effective threat response.

Fast Queries: The key to swift threat hypothesis validation

At the Black Hat US 2025 NOC, analysts used EndaceVision to rapidly validate their working hypotheses. One of the key enablers was the ability to expand query time ranges from just a few minutes to the entire conference duration (several days), without compromising on search response times. Despite the broader query window, query response times remained consistently low, allowing analysts to search through all the captured data interactively.

Analysts began their investigations by pivoting into EndaceVision™ via one of the several Endace integrations – e.g. Cisco XDR, Palo Alto Networks XSIAM (newly developed at Black Hat), Splunk etc. – and querying around the incident to analyze the incident activity. As their hypotheses evolved, analysts seamlessly expanded their search window to cover several hours, and eventually the entirety of the captured data. With the ability to filter across multiple dimensions such as IP addresses, conversations, ports, applications, protocols, etc., analysts were able to rapidly identify patterns and anomalies.

Collaboration between NOC analysts is imperative to validate hypothesis or dispense false positives. Having rapid search capabilities integrated into and across the different tool sets being using in the NOC really accelerated investigation theory sharing and investigation workflows. Rapid search into network evidence delivers contextual validation with full packet data captured before, during and after threat indicators. Having a platform that can support a large number of users with fast query times across extended look back times is a game changer for next generation NOC/SOC teams.

Total queries 2900
Median query duration 16 hours
Max query duration 1 week
Average query response time 1.5 seconds
Active analysts 20+
Top Filter dimensions IPs, Conversations, Ports, Protocol, Application, MAC, client / server
Average PCAP download time 30s
Total PCAP downloaded 53GB
Average PCAP download size 40MB
Automating PCAP Workflows for Faster Investigations

For Black Hat US 2025, Endace implemented an API that enabled integrations to extract relevant packet data (PCAP) and associated metadata, such as IP conversations, in CSV format for every incident, automatically.  Matt Vander Horst (Cisco) helped integrate this capability into Cisco XDR’s workflow automation. For each incident tracked in Cisco XDR, the system automatically triggers an API request to Endace InvestigationManager™ to extract and save the relevant PCAP and flow metadata. Links to the saved PCAP and metadata are embedded into the incident worklog for easy access. This ensured forensic data was readily available and contextually linked to the incident, eliminating the need for manual retrieval or delayed access.

SOC analysts used this capability to reduce investigation time. Instead of waiting to extract packets manually, they could immediately pivot to the saved PCAP and metadata as soon as they began reviewing an incident. This not only accelerated root cause analysis but also reduced cognitive load and fatigue, allowing analysts to focus on threat validation and response with full context.

More than 250 incidents leveraged this automation to extract and link relevant packet data, saving precious analyst time and improving overall workflow efficiency.

This integration also supported on-demand extraction of pcap and flow metadata CSV artifacts,  based on observables on the incident.

Endace InvestigationManager extracts and saves relevant PCAP for the Cisco XDR incident.

Endace InvestigationManager extracts and saves relevant PCAP for the Cisco XDR incident

The Cisco XDR worklog enrichment added links to the EndaceVision investigation, the associated PCAP, and the CSV metadata associated with the incident, to make it easy for analysts to pivot directly to the detailed forensic evidence directly from the incident worklog.

Acknowledgements

Our thanks to the Black Hat NOC team, led by Grifter, Bart, and Fink, for the opportunity to include EndaceProbes in the Black Hat NOC architecture.  Also, a special thank you to Jessica Bair Oppenheimer for including Endace in the Cisco security stack architecture for Black Hat, sharing the Cisco screens and space in the NOC.

Thanks also to Matt Vander Horst at Cisco for building the XDR integration with the Endace vault API and Aditya Sankar at Cisco for setting up automation remote access.

About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.blackhat.com.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Black Hat 2025 NOC

Original Entry by : Michael Morris

Elevating Incident Response with the Ultimate Network Forensics – PCAP Or It Didn’t Happen

By Michael Morris, Director of Global Business Development, Endace and Barry Shaw, Senior Engineering Manager, Technology Partner Programs in Fusion Integration, Endace

This year, the Black Hat NOC team was armed with a new cyber-superpower. Always-on full packet capture was deployed to record the entire conference traffic to support the Black Hat NOC/SOC directives of Protect, Educate, and Innovate.  Full packet capture provides an indelible record of all network activity, which is critical for security operations’ investigations.

Two EndaceProbes with a combined storage of 266 TB were installed in the Black Hat US 2025 NOC to capture every packet in full, from show start to the final closing. Fast access to full packet data for any event allowed the talented Black Hat NOC team to quickly understand threats and security risks for the attendees of the Black Hat conference. 62TB of captured network packet data was heavily leveraged by the Black Hat NOC security analysts with more than 1000 PCAP downloads from Endace systems during the 6 days of network operation.

Endace Fusion integrations provided the glue between the Cisco Security suite, Palo Alto Networks, Corelight and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk/Splunk Cloud, Cisco XDR, Cisco Firepower, Cisco SNA, Palo Alto Networks XSIAM, Panorama NGFWs, and Corelight Investigator through to EndaceVision and hosted Wireshark.  When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk.  Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the surprisingly persistent use of insecure protocols, and ultimately driving automation to streamline the response to these.

NOC Innovations

At the Black Hat 2025 NOC, a number of key innovations were developed and enhanced with Endace and integrations we have with Cisco, Palo Alto Networks, and Corelight. These innovations advanced and simplified the use of Endace packet data in incident response and threat hunting investigations. Endace packet data is an invaluable forensic tool for NOC/SOC analysts in getting to the root cause of complex threat investigations to be 100% sure of the impact of malicious activity.

The first integration innovation developed by Matt Vander Horst – Cisco and Barry (Baz) Shaw – Endace, and Anantha Srinivasan -Endace was a Cisco XDR automation that gathered up and preserved the packet evidence for each security alert. Links to the packet evidence and flow records CSV appear in the XDR worklog, and a link to the Endace Investigation is provided should the analyst need to investigate other related packet evidence, see screenshot below. This required new APIs on Endace, and a new workflow automation within XDR, that were both developed during the days and nights of the Black Hat NOC week.

The second integration innovation, developed by Josh Randall (#Mr Mongo) – Palo Alto Networks and Anantha  Srinivasan – Endace, created a direct pivot from any Palo Alto Networks XSIAM incident that launches an EndaceVision investigation focused on the packets related to that security event. This integration enabled analysts leveraging the power of the XSIAM SIEM platform to get directly to Endace ‘s packet-level forensics in the context of any XSIAM incident.   You can see the pivot integration in the bottom right of the Cortex XSIAM screenshot below.

The third integration innovation involved streaming packet Metadata from EndaceProbe into Splunk Cloud to create a dashboard with various insights. This included Encrypted vs Unencrypted traffic volumes, passwords in the clear, Encryption Strength, and general network traffic levels. We displayed this on a central dashboard for everyone to view during the conference, see below.

The final innovation we brought to the NOC was a change to the Endace packet search capability that resulted in a near x50 improvement in packet search and download times. With fast and easy access to full PCAP and a little instruction on how to use EndaceVision, the enthusiasm for using packet data to understand and resolve incidents really took off.

SOC Findings and Lessons Learned

The feedback we received from many NOC team members is that streamlining access to full packet data opened new possibilities for threat hunting and incident response. Fast and easy access to packet data from the other security tools in the NOC really helped our understanding of many incidents.

We found there is still too much information carried across the network in the clear, with around 8% unencrypted. This leaves users vulnerable to information leakage, credential stealing, account hijacking, social engineering and outright fraud.  The use of POP, IMAP, HTTP and other encrypted protocols is still too high.

Our biggest learning was the power of collaboration is dramatically amplified when everyone is together in a dark room, with pumping music, audio free 90s movies playing in the background, with the common goals of protect, educate and innovate! It is amazing how well everyone worked together, such collaboration between different vendors is incredible to experience and will only strengthen cybersecurity for all our users.

Acknowledgements

Our thanks to the Black Hat NOC team, led by Grifter, Bart, and Fink, for the opportunity to include EndaceProbes in the Black Hat NOC architecture.  Also, a special thank you to Jessica Bair Oppenheimer for including Endace in the Cisco security stack architecture for Black Hat, sharing the Cisco screens and space in the NOC. The NOC team includes some of the most experienced security experts across the industry. Everyone was a pleasure to work and innovate with and we came away with a great appreciation for the power of working in such a welcoming and open environment.  The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also working prototypes that were developed and proved out during the SOC.

About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.Black Hat.com.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Endace Packet Forensics Files: Episode #62

Original Entry by : Michael Morris

In Episode 62, Michael talks to Jessica (Bair) Oppenheimer, Cisco’s Director of Security Operations

By Michael Morris, Director of Global Business Development, Endace

Michael Morris, Director of Global Business Development, Endace

What does it take to run a world-class Security Operations Center (SOC) in today’s high-stakes, high-speed cybersecurity landscape?

In this episode of the Packet Forensic Files, I talk with Jessica (Bair) Oppenheimer, Cisco’s Director of Security Operations, for an in-depth look at next-generation Security Operations Centers (SOCs). Jessica shares her expertise from securing high-stakes events like the Paris 2024 Summer Olympics, the NFL Super Bowl, Black Hat, Cisco Live and RSAC Conferences, as well as years of experience at Guidance Software, ThreatGrid and Cisco.

I asked Jessica what differentiates the “next gen” SOC from traditional SOC models. She talks about some of the “high-visibility “SOCs that she’s been working with as part of the Cisco Security Operations team, and her experience bringing together highly-skilled people from multiple organizations, keeping them motivated, equipping them with all the tools and telemetry they need, so they can collaborate and innovate.

Jessica talks about why people, innovation, and smart use of AI as an enabler for SOC teams are critical to creating an agile and responsive SOC team. She sees AI’s key role is in speeding up analysis, detection, and threat-hunting, and providing analysts with detailed context around incidents so they can make informed decisions more quickly. It can also help analysts to quickly build automation tasks in response to threats they’re seeing so they can respond faster. That makes analysts more productive and effective, and in turn frees them up to do more proactive threat hunting like investigating some of the less obvious signals that might indicate potentially serious threats.

This episode is a must-watch for cybersecurity professionals who want to stay ahead of evolving threats – particularly those in security operations. It is jam-packed with insights on balancing automation with human expertise and establishing the key KPIs for SOC success.

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.

Cisco Live US 2025 SOC – PCAP or it didn’t happen!

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace

Barry

Elevating Incident Response with the Ultimate Network Forensics

After a successful SOC @ RSAC2025, our team was stoked to be invited to help Cisco run the SOC at Cisco Live US (CLUS). We jumped at the chance to work with the Cisco team again. It was a great opportunity to innovate while helping Cisco protect and educate the attendees of the conference. Plus, the Cisco team is a lot of fun to hang out with—there’s a very infectious vibe in the SOC that has everyone buzzing for the entire week.

Packet capture is essential in the SOC. It provides an indelible record of all network activity, which is invaluable to the SOC team when investigating threats or security risks—hence the phrase, “PCAP or it didn’t happen.” For CLUS, we deployed two EndaceProbes with a combined storage of 864TB to continuously record all network activity delivered via 2 x 10GbE SPAN ports. This gave us the capacity to record at least several weeks of full network packet data—covering more than the entire duration of the show.

Endace Fusion integrations provided the glue between the Cisco Security suite and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk Enterprise Security/Splunk Cloud, Cisco XDR, Cisco Firepower, and Cisco Secure Network Analytics (SNA) through to EndaceVision and hosted Wireshark. When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk. Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the heavy use of insecure protocols, and ultimately driving automation to manage the unexpected volume. File-carving was enabled, and over 750,000 files were reconstructed from packet data, with over 40,000 samples submitted to Splunk Attack Analyzer (SAA) via Endace’s automatic submission software. SAA then sent over 12,000 files to Secure Malware Analytics (formerly Threat Grid) for dynamic analysis of the behavior.

SOC Findings and Lessons Learned

The SOC team was surprised and initially overwhelmed at the volume of unencrypted traffic on the network. Logging of passwords was coupled with a Cisco XDR automation that created an incident on each detection. This resulted in a heavy workload identifying and notifying users to educate and protect them in the future. The Splunk team developed a creative automated solution to notify users that the SOC detected their use of insecure protocols.

We even found a version of POP that was news to us all—APOP. This hashes the server timestamp in the response header with the user’s password to create a password digest. While this obscures the password, it only delays its inevitable retrieval, all the while the actual message bodies are still transferred in plain text!

In the theme of plain text passwords, reviewing the connections associated with one of these sessions showed a large number of file downloads in the Zeek log generated on EndaceProbe.  This was one of many clients that used the free conference Wi-Fi to download Windows update files, but after filtering out the cab files in a Splunk search, we found a suspicious-looking file:

A search on this filename in SAA confirmed the presence of a malware download by this unfortunate user, whom the SOC team made every effort to identify.

There were also a few notable occasions where Secure Firewall alerts indicated intrusion attempts, which, after a pivot to EndaceVision, were pulled up in Wireshark for further analysis. One alert of note was a “BROWSER-IE Microsoft Internet Explorer Chakra.dll Array.filter type confusion attempt” indicating a malicious web server was trying to exploit a vulnerability in IE. 

A review of the PCAP noted that the target client was, in fact, running Safari on MacOS X.

This indicated that even if the web server was launching a legitimate attack, the client was not vulnerable to this attempt and therefore no further action was required. This highlights the value of full PCAP, packets in related sessions that don’t trigger alerts can offer valuable insight and context to security analysts. This allows rapid determinations to be made with confidence.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

Acknowledgements

Our thanks to the Cisco team, led by Jessica Bair-Oppenheimer and Steve Fink, for the opportunity to include EndaceProbes in the Cisco Live SOC architecture. The SOC team is a collection of Cisco experts across many Cisco solutions who were a pleasure to work with and innovate with.  We came away with a great appreciation for the power and ease of use of the Cisco Security tools. The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also prototype integration extensions that were developed and proved out during the SOC.

To learn more about all the ways Endace integrates with Cisco, check out:  https://www.endace.com/cisco.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/ 

Helping Protect Cisco Live 2025 in San Diego

Original Entry by : Michael Morris

By Michael Morris, Senior Director of Technology Alliances

Building on our collaboration in the SOC at RSAC 2025, Endace was honoured to be invited by Cisco to provide EndaceProbe packet capture and a team of threat hunters to support the Cisco Live Security Operations Center (SOC). Our primary goal was to protect the network and attendees at Cisco Live San Diego 2025, our secondary goals were to educate and innovate.

Two EndaceProbes were deployed in Cisco’s SOC-in-a-Box, continuously recording all network traffic provided over two 10G SPAN ports.  We integrated with Splunk Enterprise Security and Cisco Security Cloud to deliver real-time network visibility. This enabled the SOC team to detect, locate, and address threats to attendees from external and internal sources, and helped to streamline and accelerate SOC investigation workflows.

The SOC at Cisco Live aims to protect the more than 23,000 attendees from threats on the conference network. When an attendee’s device is found to be compromised or unsecured, the SOC team makes every effort to identify, locate and help remediate the issue. The SOC team also provided public tours of the SOC to educate attendees about how the SOC is configured, the processes team members implemented, and the sorts of issues and threats that we uncovered.

Drawing on our experience at RSAC 2025 (where Endace captured 36 TB of data and 45 billion packets) we further optimized our approach for Cisco Live’s SOC.

The SOC was set up in two days, leveraging prior planning and our engineers’ experience at RSAC 2025.

At Cisco Live, Endace:

  • Captured 78.9 TB of traffic representing 99.5 billion packets
  • Reassembled more than 740,000 file objects in real time and sent 42,624 files to Splunk Attack Analyzer for further analysis
  • Detected 2,256 occurrences of cleartext passwords being used from 92 unique devices
  • Provided packet-level forensic evidence to all the SOC analysts for investigating and triaging a range of security events
  • Integrated with Cisco XDR, Splunk, Cisco Firepower IDS, Splunk Attack Analyser, Cisco Secure Malware Analytics, Cisco Secure Network Analytics to provide a seamless IR workflow
  • Innovated with new integrations that streamed additional metadata into Splunk, which sped up the investigation and remediation process

It was impressive to see the whole team of Cisco and Endace Engineers pivoting to the packet data to investigate an entire range of security incidents. Inspecting the packet data enabled fast decisions because the team could see exactly what was happening on the wire before, during, and after any event. As they say, PCAP or it didn’t happen!

Having complete packet data available at the SOC team’s fingertips really highlighted the incredible value of always-on packet capture; recording every packet and delivering complete, real-time visibility into threats, performance issues, and anomalous behaviour across the network.

There is an infectious energy that we get when working in a SOC environment like this, with Engineers from different teams and companies collaborating to track down threats, and share ideas on how to improve the SOC. The Endace team came away with many learnings that will improve EndaceProbe and our partner integrations.

We look forward to being a part of the SOC again at some more big upcoming events – stay tuned, details coming soon.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/