Black Hat 2025 NOC

Original Entry by : Michael Morris

Elevating Incident Response with the Ultimate Network Forensics – PCAP Or It Didn’t Happen

By Michael Morris, Director of Global Business Development, Endace and Barry Shaw, Senior Engineering Manager, Technology Partner Programs in Fusion Integration, Endace


This year, the Black Hat NOC team was armed with a new cyber-superpower. Always-on full packet capture was deployed to record the entire conference traffic to support the Black Hat NOC/SOC directives of Protect, Educate, and Innovate.  Full packet capture provides an indelible record of all network activity, which is critical for security operations’ investigations.

Two EndaceProbes with a combined storage of 266 TB were installed in the Black Hat US 2025 NOC to capture every packet in full, from show start to the final closing. Fast access to full packet data for any event allowed the talented Black Hat NOC team to quickly understand threats and security risks for the attendees of the Black Hat conference. 62TB of captured network packet data was heavily leveraged by the Black Hat NOC security analysts with more than 1000 PCAP downloads from Endace systems during the 6 days of network operation.

Endace Fusion integrations provided the glue between the Cisco Security suite, Palo Alto Networks, Corelight and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk/Splunk Cloud, Cisco XDR, Cisco Firepower, Cisco SNA, Palo Alto Networks XSIAM, Panorama NGFWs, and Corelight Investigator through to EndaceVision and hosted Wireshark.  When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk.  Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the surprisingly persistent use of insecure protocols, and ultimately driving automation to streamline the response to these.

NOC Innovations

At the Black Hat 2025 NOC, a number of key innovations were developed and enhanced with Endace and integrations we have with Cisco, Palo Alto Networks, and Corelight. These innovations advanced and simplified the use of Endace packet data in incident response and threat hunting investigations. Endace packet data is an invaluable forensic tool for NOC/SOC analysts in getting to the root cause of complex threat investigations to be 100% sure of the impact of malicious activity.

The first integration innovation developed by Matt Vander Horst – Cisco and Barry (Baz) Shaw – Endace, and Anantha Srinivasan -Endace was a Cisco XDR automation that gathered up and preserved the packet evidence for each security alert. Links to the packet evidence and flow records CSV appear in the XDR worklog, and a link to the Endace Investigation is provided should the analyst need to investigate other related packet evidence, see screenshot below. This required new APIs on Endace, and a new workflow automation within XDR, that were both developed during the days and nights of the Black Hat NOC week.

The second integration innovation, developed by Josh Randall (#Mr Mongo) – Palo Alto Networks and Anantha  Srinivasan – Endace, created a direct pivot from any Palo Alto Networks XSIAM incident that launches an EndaceVision investigation focused on the packets related to that security event. This integration enabled analysts leveraging the power of the XSIAM SIEM platform to get directly to Endace ‘s packet-level forensics in the context of any XSIAM incident.   You can see the pivot integration in the bottom right of the Cortex XSIAM screenshot below.

The third integration innovation involved streaming packet Metadata from EndaceProbe into Splunk Cloud to create a dashboard with various insights. This included Encrypted vs Unencrypted traffic volumes, passwords in the clear, Encryption Strength, and general network traffic levels. We displayed this on a central dashboard for everyone to view during the conference, see below.

The final innovation we brought to the NOC was a change to the Endace packet search capability that resulted in a near x50 improvement in packet search and download times. With fast and easy access to full PCAP and a little instruction on how to use EndaceVision, the enthusiasm for using packet data to understand and resolve incidents really took off.

SOC Findings and Lessons Learned

The feedback we received from many NOC team members is that streamlining access to full packet data opened new possibilities for threat hunting and incident response. Fast and easy access to packet data from the other security tools in the NOC really helped our understanding of many incidents.

We found there is still too much information carried across the network in the clear, with around 8% unencrypted. This leaves users vulnerable to information leakage, credential stealing, account hijacking, social engineering and outright fraud.  The use of POP, IMAP, HTTP and other encrypted protocols is still too high.

Our biggest learning was the power of collaboration is dramatically amplified when everyone is together in a dark room, with pumping music, audio free 90s movies playing in the background, with the common goals of protect, educate and innovate! It is amazing how well everyone worked together, such collaboration between different vendors is incredible to experience and will only strengthen cybersecurity for all our users.

Acknowledgements

Our thanks to the Black Hat NOC team, led by Grifter, Bart, and Fink, for the opportunity to include EndaceProbes in the Black Hat NOC architecture.  Also, a special thank you to Jessica Bair Oppenheimer for including Endace in the Cisco security stack architecture for Black Hat, sharing the Cisco screens and space in the NOC. The NOC team includes some of the most experienced security experts across the industry. Everyone was a pleasure to work and innovate with and we came away with a great appreciation for the power of working in such a welcoming and open environment.  The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also working prototypes that were developed and proved out during the SOC.

About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.Black Hat.com.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/


Endace Packet Forensics Files: Episode #62

Original Entry by : Michael Morris

In Episode 62, Michael talks to Jessica (Bair) Oppenheimer, Cisco’s Director of Security Operations

By Michael Morris, Director of Global Business Development, Endace


Michael Morris, Director of Global Business Development, Endace

What does it take to run a world-class Security Operations Center (SOC) in today’s high-stakes, high-speed cybersecurity landscape?

In this episode of the Packet Forensic Files, I talk with Jessica (Bair) Oppenheimer, Cisco’s Director of Security Operations, for an in-depth look at next-generation Security Operations Centers (SOCs). Jessica shares her expertise from securing high-stakes events like the Paris 2024 Summer Olympics, the NFL Super Bowl, Black Hat, Cisco Live and RSAC Conferences, as well as years of experience at Guidance Software, ThreatGrid and Cisco.

I asked Jessica what differentiates the “next gen” SOC from traditional SOC models. She talks about some of the “high-visibility “SOCs that she’s been working with as part of the Cisco Security Operations team, and her experience bringing together highly-skilled people from multiple organizations, keeping them motivated, equipping them with all the tools and telemetry they need, so they can collaborate and innovate.

Jessica talks about why people, innovation, and smart use of AI as an enabler for SOC teams are critical to creating an agile and responsive SOC team. She sees AI’s key role is in speeding up analysis, detection, and threat-hunting, and providing analysts with detailed context around incidents so they can make informed decisions more quickly. It can also help analysts to quickly build automation tasks in response to threats they’re seeing so they can respond faster. That makes analysts more productive and effective, and in turn frees them up to do more proactive threat hunting like investigating some of the less obvious signals that might indicate potentially serious threats.

This episode is a must-watch for cybersecurity professionals who want to stay ahead of evolving threats – particularly those in security operations. It is jam-packed with insights on balancing automation with human expertise and establishing the key KPIs for SOC success.

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.


Cisco Live US 2025 SOC – PCAP or it didn’t happen!

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace


Barry

Elevating Incident Response with the Ultimate Network Forensics

After a successful SOC @ RSAC2025, our team was stoked to be invited to help Cisco run the SOC at Cisco Live US (CLUS). We jumped at the chance to work with the Cisco team again. It was a great opportunity to innovate while helping Cisco protect and educate the attendees of the conference. Plus, the Cisco team is a lot of fun to hang out with—there’s a very infectious vibe in the SOC that has everyone buzzing for the entire week.

Packet capture is essential in the SOC. It provides an indelible record of all network activity, which is invaluable to the SOC team when investigating threats or security risks—hence the phrase, “PCAP or it didn’t happen.” For CLUS, we deployed two EndaceProbes with a combined storage of 864TB to continuously record all network activity delivered via 2 x 10GbE SPAN ports. This gave us the capacity to record at least several weeks of full network packet data—covering more than the entire duration of the show.

Endace Fusion integrations provided the glue between the Cisco Security suite and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk Enterprise Security/Splunk Cloud, Cisco XDR, Cisco Firepower, and Cisco Secure Network Analytics (SNA) through to EndaceVision and hosted Wireshark. When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk. Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the heavy use of insecure protocols, and ultimately driving automation to manage the unexpected volume. File-carving was enabled, and over 750,000 files were reconstructed from packet data, with over 40,000 samples submitted to Splunk Attack Analyzer (SAA) via Endace’s automatic submission software. SAA then sent over 12,000 files to Secure Malware Analytics (formerly Threat Grid) for dynamic analysis of the behavior.

SOC Findings and Lessons Learned

The SOC team was surprised and initially overwhelmed at the volume of unencrypted traffic on the network. Logging of passwords was coupled with a Cisco XDR automation that created an incident on each detection. This resulted in a heavy workload identifying and notifying users to educate and protect them in the future. The Splunk team developed a creative automated solution to notify users that the SOC detected their use of insecure protocols.

We even found a version of POP that was news to us all—APOP. This hashes the server timestamp in the response header with the user’s password to create a password digest. While this obscures the password, it only delays its inevitable retrieval, all the while the actual message bodies are still transferred in plain text!

In the theme of plain text passwords, reviewing the connections associated with one of these sessions showed a large number of file downloads in the Zeek log generated on EndaceProbe.  This was one of many clients that used the free conference Wi-Fi to download Windows update files, but after filtering out the cab files in a Splunk search, we found a suspicious-looking file:

A search on this filename in SAA confirmed the presence of a malware download by this unfortunate user, whom the SOC team made every effort to identify.

There were also a few notable occasions where Secure Firewall alerts indicated intrusion attempts, which, after a pivot to EndaceVision, were pulled up in Wireshark for further analysis. One alert of note was a “BROWSER-IE Microsoft Internet Explorer Chakra.dll Array.filter type confusion attempt” indicating a malicious web server was trying to exploit a vulnerability in IE. 

A review of the PCAP noted that the target client was, in fact, running Safari on MacOS X.

This indicated that even if the web server was launching a legitimate attack, the client was not vulnerable to this attempt and therefore no further action was required. This highlights the value of full PCAP, packets in related sessions that don’t trigger alerts can offer valuable insight and context to security analysts. This allows rapid determinations to be made with confidence.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

Acknowledgements

Our thanks to the Cisco team, led by Jessica Bair-Oppenheimer and Steve Fink, for the opportunity to include EndaceProbes in the Cisco Live SOC architecture. The SOC team is a collection of Cisco experts across many Cisco solutions who were a pleasure to work with and innovate with.  We came away with a great appreciation for the power and ease of use of the Cisco Security tools. The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also prototype integration extensions that were developed and proved out during the SOC.

To learn more about all the ways Endace integrates with Cisco, check out:  https://www.endace.com/cisco.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/ 


Helping Protect Cisco Live 2025 in San Diego

Original Entry by : Michael Morris

By Michael Morris, Senior Director of Technology Alliances


Building on our collaboration in the SOC at RSAC 2025, Endace was honoured to be invited by Cisco to provide EndaceProbe packet capture and a team of threat hunters to support the Cisco Live Security Operations Center (SOC). Our primary goal was to protect the network and attendees at Cisco Live San Diego 2025, our secondary goals were to educate and innovate.

Two EndaceProbes were deployed in Cisco’s SOC-in-a-Box, continuously recording all network traffic provided over two 10G SPAN ports.  We integrated with Splunk Enterprise Security and Cisco Security Cloud to deliver real-time network visibility. This enabled the SOC team to detect, locate, and address threats to attendees from external and internal sources, and helped to streamline and accelerate SOC investigation workflows.

The SOC at Cisco Live aims to protect the more than 23,000 attendees from threats on the conference network. When an attendee’s device is found to be compromised or unsecured, the SOC team makes every effort to identify, locate and help remediate the issue. The SOC team also provided public tours of the SOC to educate attendees about how the SOC is configured, the processes team members implemented, and the sorts of issues and threats that we uncovered.

Drawing on our experience at RSAC 2025 (where Endace captured 36 TB of data and 45 billion packets) we further optimized our approach for Cisco Live’s SOC.

The SOC was set up in two days, leveraging prior planning and our engineers’ experience at RSAC 2025.

At Cisco Live, Endace:

  • Captured 78.9 TB of traffic representing 99.5 billion packets
  • Reassembled more than 740,000 file objects in real time and sent 42,624 files to Splunk Attack Analyzer for further analysis
  • Detected 2,256 occurrences of cleartext passwords being used from 92 unique devices
  • Provided packet-level forensic evidence to all the SOC analysts for investigating and triaging a range of security events
  • Integrated with Cisco XDR, Splunk, Cisco Firepower IDS, Splunk Attack Analyser, Cisco Secure Malware Analytics, Cisco Secure Network Analytics to provide a seamless IR workflow
  • Innovated with new integrations that streamed additional metadata into Splunk, which sped up the investigation and remediation process

It was impressive to see the whole team of Cisco and Endace Engineers pivoting to the packet data to investigate an entire range of security incidents. Inspecting the packet data enabled fast decisions because the team could see exactly what was happening on the wire before, during, and after any event. As they say, PCAP or it didn’t happen!

Having complete packet data available at the SOC team’s fingertips really highlighted the incredible value of always-on packet capture; recording every packet and delivering complete, real-time visibility into threats, performance issues, and anomalous behaviour across the network.

There is an infectious energy that we get when working in a SOC environment like this, with Engineers from different teams and companies collaborating to track down threats, and share ideas on how to improve the SOC. The Endace team came away with many learnings that will improve EndaceProbe and our partner integrations.

We look forward to being a part of the SOC again at some more big upcoming events – stay tuned, details coming soon.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/ 


Endace Packet Forensics Files: Episode #61

Original Entry by : Michael Morris

In Episode 61, Michael talks to JP Bergeaux, Federal CTO at GuidePoint Security

By Michael Morris, Director of Global Business Development, Endace


Michael Morris, Director of Global Business Development, Endace

In my latest episode of the Endace Packet Forensic Files, I sit down with Jean-Paul (JP) Bergeaux, Federal CTO for GuidePoint Security, to explore federal cybersecurity. Our conversation dives into the challenges, technologies, and approaches reshaping how government agencies protect their digital infrastructure.

The critical importance of certifications like FIPS 140 and NIAP becomes clear. These aren’t bureaucratic checkboxes–they’re safeguards that ensure the reliability and security of technological solutions across federal networks. JP’s insights show how these standards help maintain the integrity of government systems.

The M-21-31 directives also emerge as a game-changer. Introduced in repsonse to the SolarWinds breach, these guidelines are transforming how agencies approach network forensics. Packet capture (PCAP) data is now considered the gold standard for threat detection, providing what JP calls “ground truth” in cybersecurity investigations. The real-world examples he shares are particularly compelling, especially cases where PCAP data reveals hidden threats.

We also tackle the challenges posed by generative AI. JP describes the “generative AI arms race”, where threat actors innovate rapidly, while government agencies must proceed with caution. It’s a balance between innovation and security that will define cybersecurity’s future.

One thing is clear from our conversation: the federal cybersecurity landscape is dynamic and demanding. Reactive security models are giving way to proactive approaches that integrate security across every layer of infrastructure.

Don’t miss this episode as JP shared valuable  insights into the front lines of federal cybersecurity and the tools, policies, and mindsets needed to stay ahead.

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.


Endace Packet Forensics Files: Episode #60

Original Entry by : Michael Morris

In our 60th Episode, Michael talks to James Spiteri, Director of Product Management for Security Analytics at Elastic

By Michael Morris, Director of Global Business Development, Endace


Michael Morris, Director of Global Business Development, Endace

It’s my pleasure to welcome James Spiteri from Elastic for this 60th Episode of the Packet Forensics Files. It’s a great milestone to have reached, and the series continues to grow in popularity – thanks to people like James who have joined me to share their valuable expertise and advice.

In this episode James brings a wealth of experience, having worked in cybersecurity and security operations for many years. From leading SOC teams to developing advanced solutions for generative AI and machine learning, his expertise is second to none.

We dive into the evolving landscape of nation-state cybersecurity threats. According to James, these attacks are highly sophisticated, leveraging bespoke malware, supply chain compromises, and cloud infrastructure. For SIEM vendors, this means platforms must provide comprehensive visibility and support diverse data sources to detect these threats effectively. Modern techniques like entity analytics, user behavior monitoring, and generative AI are essential in addressing these challenges.

Evolving cybersecurity regulations like GDPR and DORA demand effective data management and integrity. James highlights the role of AI in simplifying these processes, from validating data to automating complex tasks like incident reporting. Additionally, integrating SIEMs with legacy systems in critical infrastructure requires creative solutions, such as monitoring network events around outdated devices, to maintain visibility.

As we look to the future, James underscores the transformative role of generative AI in cybersecurity, both as a tool for defending against attacks and a potential weapon in the hands of cybercriminals. By staying ahead of these trends and embracing innovation, SIEM vendors can ensure organizations are better equipped to tackle the sophisticated threats of tomorrow.

Don’t miss this essential conversation—tune in for expert insights on how to fortify your defenses in the face of an increasingly complex cyber landscape.

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.


Endace Packet Forensics Files: Episode #59

Original Entry by : Michael Morris

Michael talks to Matt Bromiley about the importance of packet capture in threat hunting and how AI can improve detection and response.

By Michael Morris, Director of Global Business Development, Endace


Michael Morris, Director of Global Business Development, Endace

With limited network visibility and overwhelming data volumes, organizations struggle to detect and respond to advanced cyber threats.  

In this episode of the Endace Packet Forensics Files, I talk with Matt Bromiley, a veteran in threat hunting and incident response. With over a decade of experience and a role as a SANS instructor, Matt brings a wealth of practical knowledge to our discussion.

Matt highlights the importance of robust detection and response systems before beginning any threat hunt. He explains that even when a hunt doesn’t yield immediate results, the insights gained are invaluable for understanding the security landscape. Matt points out that proactive threat hunting is about deeply understanding network traffic, which offers significant advantages over more traditional reactive approaches.

During our conversation, Matt emphasised network packet data’s critical role in cybersecurity. He describes it as the “glue” that ties together various pieces of evidence, providing a comprehensive view of any potential attack. According to Matt, analyzing decrypted traffic and DNS logs is essential for uncovering hidden threats that might remain undetected.

Matt talks about the challenges of threat hunting, particularly when dealing with large volumes of packet data and navigating legal constraints. He stresses the necessity of having a skilled team and the right tools to manage these challenges effectively. He also shares his insights on the growing role of AI in threat hunting, predicting that it will increasingly help automate routine tasks, freeing up analysts to focus on more complex threats.

Matt’s expertise underscores the importance of a proactive approach, a deep understanding of network data, and the use of the right tools to stay ahead of cyber threats.

Don’t miss this insightful episode, where Matt provides actionable advice for enhancing your threat-hunting capabilities and strengthening your cybersecurity defenses.  

Follow Matt on Linkedin

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.


Endace Packet Forensics Files: Episode #57

Original Entry by : Michael Morris

Michael talks to Ryan Chapman about the growing complexity of ransomware – how to prepare, investigate and respond.

By Michael Morris, Director of Global Business Development, Endace


Michael Morris, Director of Global Business Development, Endace

Ransomware has shifted from simple, isolated attacks to coordinated, human-operated campaigns that target entire organizations.  

In this episode of the Endace Packet Forensics Files, I talk with Ryan Chapman, SANS Instructor and expert in Digital Forensic and Incident Response (DFIR) about these evolving threats.  

Ryan explains how attackers are becoming more methodical and sophisticated, focusing on disabling EDR/XDR solutions to evade detection and leaving organizations vulnerable to advanced attacks.  

One of the key challenges Ryan highlights is visibility. Without robust logging, packet capture, and monitoring tools, it’s nearly impossible to understand how an attack happened fully. Even encrypted traffic can reveal critical patterns if analyzed properly.   

Ryan shares examples of organizations that suffered reinfections because they rushed to restore systems without identifying the original entry point. Packet capture data plays a vital role in pinpointing when and how attackers infiltrated, ensuring a safe recovery and minimizing disruption.  

As ransomware tactics evolve, adopting a Zero-Trust approach is essential. Ryan discusses how limiting permissions and avoiding overly trusting software configurations can help prevent breaches. He cites the Kaseya attack, where some organizations avoided compromise by not blindly whitelisting trusted directories. As attackers increasingly use legitimate tools, verifying all network activity and following least privilege principles are critical defenses.   

Don’t miss this insightful episode, where Ryan provides actionable advice for preparing your organization against today’s ransomware threats.  

Follow Ryan on Linkedin

Other episodes in the Secure Networks video/audio podcast series are available here. Or listen to the podcast here or on your favorite podcast platform.