Category: Black Hat NOC

Accelerating Security Operations at Black Hat USA 2025

Original Entry by : Anantha Srinivasan

Fast Queries and automated PCAP Workflows

By Anantha Srinivasan, Senior Software Manager, Endace and Sundarram Paravastu, Principal Software Engineer, Endace

In the fast-paced environment of Security Operations Centers (SOCs), swift incident response and threat hunting are essential to combat sophisticated cyber threats. Slow queries, siloed workflows, and manual PCAP retrieval significantly hinder SOC operations, delaying investigations and prolonging detection and remediation times. These inefficiencies, compounded by limited integration and automation, exacerbate analyst fatigue and increase the risk of overlooking critical threats.

At the Black Hat US 2025 NOC, several key innovations were developed and enhanced with Endace and the integrations we have with Cisco, Palo Alto Networks and Corelight. Read: Endace Always-on Packet Capture for additional details.

The innovations developed at Black Hat facilitated the use of Endace’s always-on packet data in incident response and threat hunting workflows, empowering analysts to leverage deep forensic insights more quickly and intuitively during investigations.

Meta-data, flow data or partial packet capture can be useful in detecting possible threats and enabling analysts to get an overview of threat activity. But it often doesn’t contain enough detail for analysts to be certain about exactly what happened.

Always-on packet capture ensures analysts have a complete record of all network activity, giving them access to definitive evidence for threat investigations. Not only can they investigate specific events, but they can also look at what else happened before, after or during the specific event they are looking into.

This lets analysts build a complete picture of threat activity that is otherwise difficult or impossible without access to complete packet evidence. With detailed forensic evidence at their fingertips, they can make better decisions more quickly, speeding investigations and enabling more effective threat response.

Fast Queries: The key to swift threat hypothesis validation

At the Black Hat US 2025 NOC, analysts used EndaceVision to rapidly validate their working hypotheses. One of the key enablers was the ability to expand query time ranges from just a few minutes to the entire conference duration (several days), without compromising on search response times. Despite the broader query window, query response times remained consistently low, allowing analysts to search through all the captured data interactively.

Analysts began their investigations by pivoting into EndaceVision™ via one of the several Endace integrations – e.g. Cisco XDR, Palo Alto Networks XSIAM (newly developed at Black Hat), Splunk etc. – and querying around the incident to analyze the incident activity. As their hypotheses evolved, analysts seamlessly expanded their search window to cover several hours, and eventually the entirety of the captured data. With the ability to filter across multiple dimensions such as IP addresses, conversations, ports, applications, protocols, etc., analysts were able to rapidly identify patterns and anomalies.

Collaboration between NOC analysts is imperative to validate hypothesis or dispense false positives. Having rapid search capabilities integrated into and across the different tool sets being using in the NOC really accelerated investigation theory sharing and investigation workflows. Rapid search into network evidence delivers contextual validation with full packet data captured before, during and after threat indicators. Having a platform that can support a large number of users with fast query times across extended look back times is a game changer for next generation NOC/SOC teams.

Total queries 2900
Median query duration 16 hours
Max query duration 1 week
Average query response time 1.5 seconds
Active analysts 20+
Top Filter dimensions IPs, Conversations, Ports, Protocol, Application, MAC, client / server
Average PCAP download time 30s
Total PCAP downloaded 53GB
Average PCAP download size 40MB
Automating PCAP Workflows for Faster Investigations

For Black Hat US 2025, Endace implemented an API that enabled integrations to extract relevant packet data (PCAP) and associated metadata, such as IP conversations, in CSV format for every incident, automatically.  Matt Vander Horst (Cisco) helped integrate this capability into Cisco XDR’s workflow automation. For each incident tracked in Cisco XDR, the system automatically triggers an API request to Endace InvestigationManager™ to extract and save the relevant PCAP and flow metadata. Links to the saved PCAP and metadata are embedded into the incident worklog for easy access. This ensured forensic data was readily available and contextually linked to the incident, eliminating the need for manual retrieval or delayed access.

SOC analysts used this capability to reduce investigation time. Instead of waiting to extract packets manually, they could immediately pivot to the saved PCAP and metadata as soon as they began reviewing an incident. This not only accelerated root cause analysis but also reduced cognitive load and fatigue, allowing analysts to focus on threat validation and response with full context.

More than 250 incidents leveraged this automation to extract and link relevant packet data, saving precious analyst time and improving overall workflow efficiency.

This integration also supported on-demand extraction of pcap and flow metadata CSV artifacts,  based on observables on the incident.

Endace InvestigationManager extracts and saves relevant PCAP for the Cisco XDR incident.

Endace InvestigationManager extracts and saves relevant PCAP for the Cisco XDR incident

The Cisco XDR worklog enrichment added links to the EndaceVision investigation, the associated PCAP, and the CSV metadata associated with the incident, to make it easy for analysts to pivot directly to the detailed forensic evidence directly from the incident worklog.

Acknowledgements

Our thanks to the Black Hat NOC team, led by Grifter, Bart, and Fink, for the opportunity to include EndaceProbes in the Black Hat NOC architecture.  Also, a special thank you to Jessica Bair Oppenheimer for including Endace in the Cisco security stack architecture for Black Hat, sharing the Cisco screens and space in the NOC.

Thanks also to Matt Vander Horst at Cisco for building the XDR integration with the Endace vault API and Aditya Sankar at Cisco for setting up automation remote access.

About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.blackhat.com.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Black Hat 2025 NOC

Original Entry by : Michael Morris

Elevating Incident Response with the Ultimate Network Forensics – PCAP Or It Didn’t Happen

By Michael Morris, Director of Global Business Development, Endace and Barry Shaw, Senior Engineering Manager, Technology Partner Programs in Fusion Integration, Endace

This year, the Black Hat NOC team was armed with a new cyber-superpower. Always-on full packet capture was deployed to record the entire conference traffic to support the Black Hat NOC/SOC directives of Protect, Educate, and Innovate.  Full packet capture provides an indelible record of all network activity, which is critical for security operations’ investigations.

Two EndaceProbes with a combined storage of 266 TB were installed in the Black Hat US 2025 NOC to capture every packet in full, from show start to the final closing. Fast access to full packet data for any event allowed the talented Black Hat NOC team to quickly understand threats and security risks for the attendees of the Black Hat conference. 62TB of captured network packet data was heavily leveraged by the Black Hat NOC security analysts with more than 1000 PCAP downloads from Endace systems during the 6 days of network operation.

Endace Fusion integrations provided the glue between the Cisco Security suite, Palo Alto Networks, Corelight and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk/Splunk Cloud, Cisco XDR, Cisco Firepower, Cisco SNA, Palo Alto Networks XSIAM, Panorama NGFWs, and Corelight Investigator through to EndaceVision and hosted Wireshark.  When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk.  Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the surprisingly persistent use of insecure protocols, and ultimately driving automation to streamline the response to these.

NOC Innovations

At the Black Hat 2025 NOC, a number of key innovations were developed and enhanced with Endace and integrations we have with Cisco, Palo Alto Networks, and Corelight. These innovations advanced and simplified the use of Endace packet data in incident response and threat hunting investigations. Endace packet data is an invaluable forensic tool for NOC/SOC analysts in getting to the root cause of complex threat investigations to be 100% sure of the impact of malicious activity.

The first integration innovation developed by Matt Vander Horst – Cisco and Barry (Baz) Shaw – Endace, and Anantha Srinivasan -Endace was a Cisco XDR automation that gathered up and preserved the packet evidence for each security alert. Links to the packet evidence and flow records CSV appear in the XDR worklog, and a link to the Endace Investigation is provided should the analyst need to investigate other related packet evidence, see screenshot below. This required new APIs on Endace, and a new workflow automation within XDR, that were both developed during the days and nights of the Black Hat NOC week.

The second integration innovation, developed by Josh Randall (#Mr Mongo) – Palo Alto Networks and Anantha  Srinivasan – Endace, created a direct pivot from any Palo Alto Networks XSIAM incident that launches an EndaceVision investigation focused on the packets related to that security event. This integration enabled analysts leveraging the power of the XSIAM SIEM platform to get directly to Endace ‘s packet-level forensics in the context of any XSIAM incident.   You can see the pivot integration in the bottom right of the Cortex XSIAM screenshot below.

The third integration innovation involved streaming packet Metadata from EndaceProbe into Splunk Cloud to create a dashboard with various insights. This included Encrypted vs Unencrypted traffic volumes, passwords in the clear, Encryption Strength, and general network traffic levels. We displayed this on a central dashboard for everyone to view during the conference, see below.

The final innovation we brought to the NOC was a change to the Endace packet search capability that resulted in a near x50 improvement in packet search and download times. With fast and easy access to full PCAP and a little instruction on how to use EndaceVision, the enthusiasm for using packet data to understand and resolve incidents really took off.

SOC Findings and Lessons Learned

The feedback we received from many NOC team members is that streamlining access to full packet data opened new possibilities for threat hunting and incident response. Fast and easy access to packet data from the other security tools in the NOC really helped our understanding of many incidents.

We found there is still too much information carried across the network in the clear, with around 8% unencrypted. This leaves users vulnerable to information leakage, credential stealing, account hijacking, social engineering and outright fraud.  The use of POP, IMAP, HTTP and other encrypted protocols is still too high.

Our biggest learning was the power of collaboration is dramatically amplified when everyone is together in a dark room, with pumping music, audio free 90s movies playing in the background, with the common goals of protect, educate and innovate! It is amazing how well everyone worked together, such collaboration between different vendors is incredible to experience and will only strengthen cybersecurity for all our users.

Acknowledgements

Our thanks to the Black Hat NOC team, led by Grifter, Bart, and Fink, for the opportunity to include EndaceProbes in the Black Hat NOC architecture.  Also, a special thank you to Jessica Bair Oppenheimer for including Endace in the Cisco security stack architecture for Black Hat, sharing the Cisco screens and space in the NOC. The NOC team includes some of the most experienced security experts across the industry. Everyone was a pleasure to work and innovate with and we came away with a great appreciation for the power of working in such a welcoming and open environment.  The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also working prototypes that were developed and proved out during the SOC.

About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.Black Hat.com.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/