By Cary Wright, VP Product, Endace

---

Why IPv6 Matters

IPv4’s roughly 4.3 billion available addresses are nowhere near enough for today’s hyper‑connected world. In contrast, IPv6 offers a 128‑bit address space, providing 3.4 x 10³⁸ possible addresses — more than enough for decades of growth.  Technologies like Network Address Translation (NAT), private addressing, and CIDR have extended IPv4 far beyond its natural lifespan. These workarounds give organizations a false sense that IPv4 is still sufficient, reducing the urgency to adopt IPv6.

What we observed in Amsterdam

In the Cisco Live EMEA SOC, we inspected 130 billion packets across 32,434 unique IP endpoint devices at the conference, using Splunk to query unique DHCP Client IDs to measure. These included devices connecting to the Wi-Fi and wired networks at the Cisco Live conference network, including attendee laptops, phones, and conference devices such as demo stations, cameras, IOT devices, displays, networking equipment, and any other IP connected device.

Of this traffic, 62% of the data travelled over IPv6, and only 38% over IPv4. This represents a tectonic shift in the move to IPv6. Perhaps this was because we were sitting just a few miles from the Regional Internet Registry for Europe, Middle East and Central Asia (RIPE NCC), or more likely this is because the world is finally ready and moving to IPv6.

Our heaviest day was Tuesday, with 25,609 devices that connected to the network.

Across all this traffic we observed 1.7 million unique IP addresses, most of which were external addresses accessed by attendees and conference devices. Those IP addresses were made up of 386,397 IPv4 addresses, and 1,339,329 IPv6 addresses.

Threat Actors — adopting IPv6 faster than anyone

In the SOC, we have no shortage of data to interrogate, interrogating our Splunk data highlight that threat actors are now heavily favoring IPv6 to conduct their attacks, hijack resources, or compromise systems.  Over 99% of malicious URLs and crypto miners used IPv6, telling us that we need to ensure we properly secure our IPv6 infrastructure. Just 1% of our attacks involved IPv4. That indicates a trend that we all need to take notice of.

A Steady Shift — But an Inevitable One

Although the transition has taken decades, IPv6 momentum appears to have crossed an important threshold. With increasing digital demands, rising IPv4 costs, and rapidly expanding device ecosystems, IPv6 isn’t just beneficial — it’s essential.

The future of the Internet is unquestionably IPv6. The challenge now is how quickly the world can get there, and how well we secure it. At Cisco Live EMEA, we saw the world has taken a large and important step forward.

Acknowledgements

This important insight to IPv6 adoption would not have been possible without the great work done by the Cisco Live EMEA SOC team, led by Jessica Oppenheimer and Ivan Berlinson.

Data collected and analyzed was the result of a team, many thanks go to the following team members:

Network Operations Center Liaisons

• Remco Kamerman, Luke Hebditch, Mark Bremner and Scott Neuman

Cisco Security and Splunk SOC Team

• SOC in a Box: Adi Sankar
• Splunk Security Integrations: Paul Pelletier and Kenneth Bouchard, with Josh Wilson and Duane Waddle
• Splunk Threat Researchers: Nasreddine Bencherchali and Paul Pang
• Breach Protection Suite: Mark Pleunes, Ibrahim Yusuf, Piotr Jarzynka, Matt Vander Horst, Yannis Steiakogiannakis and Eric Rennie, with Bilal Qamar
• User Protection Suite: Aaron Woland
• Firewall and Security Cloud Control: Adam Kilgore and Christopher Grabowski

Endace SOC Team

• Co-SOC Leader: Cary Wright
• Endace Engineering: Owen Gallagher, Sundarram Paravastu and Sam Brockelsby

Read related Cisco Team Blogs from the Cisco Live Europe 2026 SOC: 
https://blogs.cisco.com/security/emea-soc-2026

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

Event SOC Website 
Visit Cisco’s Event SOC website for full details of the SOC setup, and download the whitepaper written by Jessica Oppenheimer:
https://www.cisco.com/site/us/en/products/security/event-soc-report.html

By Owen Gallagher, Senior Sales Engineer, Endace

---

Overview

This year, I had the opportunity to work in a Security Operations Centre (SOC) for the first time in over a decade, at Cisco® Live Amsterdam. I joined the team responsible for monitoring and responding to incidents on the public Wi‑Fi network.

Cisco builds a fully operational SOC at each Cisco Live event, using tools from Cisco, Splunk and Endace to secure the environment. The team is a mix of engineers and analysts from all three companies, with experience ranging from first‑time responders to L3 (also known as Tier 3) analysts. Even though many of us had never worked together before, the teamwork was excellent. Everyone shared knowledge, helped each other understand different technologies and supported one another during investigations.

Because this was my first SOC shift in over 10 years, it took a little time to get into the rhythm of moving from a Cisco XDR Incident notification, gathering evidence, and deciding whether to close or escalate an incident. But after working through a full investigation, things started to click again.

One incident in particular stood out, so I’ll walk through the workflow and how we reached our conclusion.

The Notification

Cisco XDR® reported suspicious activity: a suspicious IP response from an external host. My first task was to understand what this external IP address was and which internal devices it had communicated with on the Wi‑Fi network. The screenshots below show how the alert appeared in Cisco XDR, including the details provided in the incident description.

Identifying the External IP

I used Talos® Intelligence to look up the IP address. Talos showed that the IP was untrusted and already on the block list, which immediately made the incident more important to investigate.

Checking Internal Communication

Next, I moved into Splunk®. By filtering logs from the Cisco Secure Firewall® (running in passive mode), I could see that the suspicious IP had communicated twice with two internal assets. Based on the address range, these devices were on the wireless network.

The logs also showed that, if the firewall had been in active mode, this traffic would have been blocked. The protocol involved was ICMP, meaning these were ping requests.

At this point, I knew what happened and which devices were involved. But I still needed to confirm whether the pings were successful and what the actual packets looked like.

Verifying with Packet Capture

To answer those questions, I turned to Endace, which records all network traffic. Using EndaceVision™, I generated a visualization that showed two 64‑byte packets exchanged between the external IP and the two internal devices. This matched what Splunk was showing.

To dig deeper, I pivoted into Wireshark™, hosted on EndaceProbe™, to inspect the raw packets. Wireshark confirmed two‑way ICMP communication, which gave us the evidence we needed to document the incident accurately

Escalation

I gathered screenshots, packet details and log information and documented in my report in the  XDR Incident Worklog. Because this was confirmed communication with an untrusted IP, I escalated the incident to the network team for a perimeter firewall review and to ensure that the IP address was added to the block list. 

Final Thoughts

This incident reminded me how important full packet capture is. Logs show that something happened, but packets tell you exactly how it happened and give you the confidence to take the right action.

Working in the Cisco Live SOC after so many years away from this environment was a rewarding experience. The collaboration, the technology and the live investigations made it a great week, and it reinforced why SOC work is so valuable: you get to protect real users in real time and understand the network at a level you can’t get anywhere else.

Acknowledgements

My experience at this event would not have been possible without the great work done by the Cisco Live EMEA SOC team, led by Jessica Oppenheimer and Ivan Berlinson.

Data collected and analyzed was the result of a team, many thanks go to the following team members:

Network Operations Center Liaisons

• Remco Kamerman, Luke Hebditch, Mark Bremner and Scott Neuman

Cisco Security and Splunk SOC Team

• SOC in a Box: Adi Sankar
• Splunk Security Integrations: Paul Pelletier and Kenneth Bouchard, with Josh Wilson and Duane Waddle
• Splunk Threat Researchers: Nasreddine Bencherchali and Paul Pang
• Breach Protection Suite: Mark Pleunes, Ibrahim Yusuf, Piotr Jarzynka, Matt Vander Horst, Yannis Steiakogiannakis and Eric Rennie, with Bilal Qamar
• User Protection Suite: Aaron Woland
• Firewall and Security Cloud Control: Adam Kilgore and Christopher Grabowski

Endace SOC Team

• Co-SOC Leader: Cary Wright
• Endace Engineering: Owen Gallagher, Sundarram Paravastu and Sam Brockelsby

Read related Cisco Team Blogs from the Cisco Live Europe 2026 SOC: 
https://blogs.cisco.com/security/emea-soc-2026

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

Event SOC Website 
Visit Cisco’s Event SOC website for full details of the SOC setup, and download the whitepaper written by Jessica Oppenheimer:
https://www.cisco.com/site/us/en/products/security/event-soc-report.html

Sundarram Paravastu, Principal Software Engineer, Endace

---

Overview

At each Security Operations Center (SOC) event, we capture and inspect every packet from start of show through to the very last hour, for the purpose of securing the attendees and conference, wiping the data at the end.

We had two of Endace’s newest EP94C8 EndaceProbes™ doing full packet capture providing unique insight into all activity on the network, delivering critical context and evidence for Incident Response and Threat Hunting teams. As part of the setup, we also had Application Dock VMs running on the EndaceProbes, extracting files where possible from unencrypted traffic, and sending rich metadata and Zeek® logs to Splunk® and Splunk Attack Analyzer®.

Stats	Count
Total files extracted	1,743,259
Total files submitted for malware analysis	55,939
Top extensions of files submitted	.xz, .gz, .pptx, .zip, .vnd, .pdf
Files extracted from email (POP3, IMAP etc.)	887

As part of SOC, we extracted more than 1.7M files and many of them were filtered out based on a known benign file list or excluded if they were duplicates of files already submitted. Within the files that were submitted, there were many files that were extracted from HTTP, POP3, IMAP etc.

Incident example: Malicious trojan file sent over email to compromise user system

There were 70+ users using POP3 emails and all their communication was in the clear. It’s worth noting that the SOC already has Splunk automation in place (using logs submitted to Splunk by EndaceProbe-hosted VMs) that notify affected users via email that their communications are using protocols (like POP3, IMAP) that expose their credentials.

In this specific incident example, we went looking for .rar files that were extracted and submitted to Splunk Attack Analyzer. Malicious .rar files have some notoriety so it was natural to look for any potential anomalies. As it turned out, there was indeed an email attachment that had malicious files in it.

The example above had a trojan file inside the archive, that was flagged as malicious.

We now had to trace that file extracted back to the user and their IP addresses for further analysis. Looking into Splunk’s file submission log, we could narrow down the hosts responsible for sending the email:

Digging further using the packet data recorded by our EndaceProbes, we were able to access all that affected user’s POP3 traffic over the two days of the conference and review it.

We analyzed the actual PCAP data related to these POP3 sessions using Wireshark™ – hosted on EndaceProbes – to find the user related to the event.

We then extracted all the files related to those POP3 sessions so they could be submitted to Splunk Attack Analyzer for further analysis.

We did not find further malicious files sent to the user other than the one we had originally found. However, some of the links in the emails were flagged as suspicious.

In this incident, we not only analyzed the files extracted at the time of capture but were also able to retrospectively get all related packets and extract all files related to the host involved in this incident. This enabled us to do very comprehensive, and conclusive, incident analysis.

Acknowledgements

This would not have been possible without the great work done by the Cisco Live EMEA SOC team, led by Jessica Oppenheimer and Ivan Berlinson.

Data collected and analyzed was the result of a team, many thanks go to the following team members:

Network Operations Center Liaisons

• Remco Kamerman, Luke Hebditch, Mark Bremner and Scott Neuman

Cisco Security and Splunk SOC Team

• SOC in a Box: Adi Sankar
• Splunk Security Integrations: Paul Pelletier and Kenneth Bouchard, with Josh Wilson and Duane Waddle
• Splunk Threat Researchers: Nasreddine Bencherchali and Paul Pang
• Breach Protection Suite: Mark Pleunes, Ibrahim Yusuf, Piotr Jarzynka, Matt Vander Horst, Yannis Steiakogiannakis and Eric Rennie, with Bilal Qamar
• User Protection Suite: Aaron Woland
• Firewall and Security Cloud Control: Adam Kilgore and Christopher Grabowski

Endace SOC Team

• Co-SOC Leader: Cary Wright
• Endace Engineering: Owen Gallagher, Sundarram Paravastu and Sam Brockelsby

Read related Cisco Team Blogs from the Cisco Live Europe 2026 SOC: 
https://blogs.cisco.com/security/emea-soc-2026

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

Event SOC Website 
Visit Cisco’s Event SOC website for full details of the SOC setup, and download the whitepaper written by Jessica Oppenheimer:
https://www.cisco.com/site/us/en/products/security/event-soc-report.html

By Sam Brockelsby, Senior Software Engineer, Endace

---

Overview

This year, I got my first taste of what it’s like to be part of the Security Operations Center (SOC) team working at Cisco Live® Europe 2026 in Amsterdam. The SOC team is responsible for analyzing and determining the appropriate response for various security incidents on the public Wi-Fi network for the Cisco Live event. This includes everything from relatively harmless port scanning to command-and-control attempts from malicious entities.

Coming into the SOC as a newbie was exciting and a little bit daunting. There were a lot of people to meet and a lot of new things to learn. At first, I found myself a little overwhelmed getting to grips with all the various tools, technologies and processes – fortunately, everyone on the team was incredibility supportive, open to questions and eager to help out.
As a result, I was able to get up and running within a couple of hours and escalated my first incident before lunch on the first day. I will walk through this incident in the next section.

Incident Example: Authentication Bypass Attempt

Cisco XDR® had raised an incident for an Authentication Bypass Attempt. XDR is a tool that was used in the SOC to report incidents and provide a structure to the subsequent investigation. It has powerful automated features that allow an analyst to quickly pivot into other tools such as Splunk® and EndaceProbe™. Think of it as the glue that pulls all the various tools we used in the SOC together.

In my example, the incident was detected by the Cisco Secure Firewall® via Splunk SIEM. This incident involved two endpoints on the network that were communicating with several other devices.

The detail provided by XDR told me that a number of these Authentication Bypass events had occurred since the start of the day. It also told me that the devices were all PTZOptics VHD PTZ Cameras. This suggested that there were camera systems at the event that were vulnerable.

The next step was to look at the packets. From XDR I was able to click on a Pivot-to-Vision™ link to our EndaceProbes for one of these events. Two EndaceProbe appliances captured all the traffic on the public Wi-Fi network for the event – more than 120TB of data across four days. This is invaluable for the SOC, since it guarantees that an analyst can always go back and look at the exact packet data for any given security incident.

The Pivot-to-Vision link generated an EndaceVision™ investigation on the EndaceProbes with an IP filter for the endpoint I had selected in XDR, for a 10 minute time window around the detected event in question.

EndaceVision has various tools to help understand the nature of captured traffic. In my case the Traffic Breakdown chart showed that there was a lot of http_video traffic present in my selection. This was of no interest to me, so I added an application filter to omit it from my investigation. I then launched hosted Wireshark™.

In Wireshark I applied some filters to search for POST requests to the login endpoint. I then inspected the resulting conversations between the endpoint and the other devices on the recording subnet. The conversations were unencrypted over HTTP and clear text credentials were visible.

At the very least then, we had unencrypted traffic on the network that exposed login credentials. In addition to this though, the username and password looked a bit odd to me, so I did a quick Google search on the PTZOptics VHD PTZ Camera. It turns out that these cameras have a CVE where they come configured with default hardcoded credentials that cannot be modified by the user. These default credentials can be easily found by anyone on the web, and so even if the camera operators fixed the unencrypted traffic problem, they would still be vulnerable.

The final step in the process was to escalate the issue up the chain to the Network Operations Center, with the recommendation that the camera operator take steps to encrypt the traffic and update firmware to the latest version, which patches the CVE.

Final Thoughts

Working in the SOC is a collaborative process. For this incident (and others I worked on), I needed to ask for help on numerous occasions. The SOC team we had in Amsterdam was a diverse group of people with expertise in different areas, and each person brought something unique to the table. No one has every answer to every problem, so it’s a given that you will need to ask someone else to give you a hand at some point.

For that collaboration to work though, you need to have the right kind of environment – one that challenges people to do their best, while at the same time encouraging people to speak up and ask questions free from judgement. In my experience, we got that balance right.

Acknowledgements

My experience at this event would not have been possible without the great work done by the Cisco Live EMEA SOC team, led by Jessica Oppenheimer and Ivan Berlinson.

Data collected and analyzed was the result of a team, many thanks go to the following team members:

Network Operations Center Liaisons

• Remco Kamerman, Luke Hebditch, Mark Bremner and Scott Neuman

Cisco Security and Splunk SOC Team

• SOC in a Box: Adi Sankar
• Splunk Security Integrations: Paul Pelletier and Kenneth Bouchard, with Josh Wilson and Duane Waddle
• Splunk Threat Researchers: Nasreddine Bencherchali and Paul Pang
• Breach Protection Suite: Mark Pleunes, Ibrahim Yusuf, Piotr Jarzynka, Matt Vander Horst, Yannis Steiakogiannakis and Eric Rennie, with Bilal Qamar
• User Protection Suite: Aaron Woland
• Firewall and Security Cloud Control: Adam Kilgore and Christopher Grabowski

Endace SOC Team

• Co-SOC Leader: Cary Wright
• Endace Engineering: Owen Gallagher, Sundarram Paravastu and Sam Brockelsby

Read related Cisco Team Blogs from the Cisco Live Europe 2026 SOC: 
https://blogs.cisco.com/security/emea-soc-2026

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

Event SOC Website 
Visit Cisco’s Event SOC website for full details of the SOC setup, and download the whitepaper written by Jessica Oppenheimer:
https://www.cisco.com/site/us/en/products/security/event-soc-report.html

By Cary Wright, VP Product, Endace

---

Diverse Dataset over 5 Major Conferences

Over the five deployments the SOC architecture was subject to a variety of traffic in North America, Asia, and Australia, with attendees representing most regions. Some interesting stats from what we saw:

Attendees	109,500 (over 5 conferences)
Packets Captured (TB)	204.8 Terabytes (236 Billion packets)
Unique Hosts	129,021
Sessions	2.775 (Billion)
Files Extracted by Endace	1,461,000
Files submitted to Splunk Attack Analyzer	86,000
Files submitted to Secure Malware Analytics	24,700
Password in the clear events	9,527
Devices with Password in the clear	291
Logs sent to Splunk (M)	6.75 Billion
DNS requests	428 Million
Encrypted traffic	82%

A Wide Variety of Threats

We’ve investigated and responded to a wide variety of threats, from simple passwords in the clear, to beaconing, RATs, port scanning, owned hosts, infected files, insecure applications, AI generated malicious domains, potential APTs obfuscating their C2 communications, exploits of known vulnerabilities and new novel threats.

There were also a bunch of false positives that we needed to run down. With Endace continuous packet capture integrated with the Cisco security stack we were able to dig deep to understand even the most challenging threats. By recording every packet from start of show to the very last moments we could arm the analysts with the evidence they needed to hunt down all manner of threats, if we were only capturing based on triggers or events we would have missed many of the threats that we did discover.

A great example of a threat we identified and responded to is captured by Daniel Lawson’s blog: Endace Full Packet Capture finds Active Directory Credentials in Clear Text.

For these cybersecurity conferences our environment needed to be more permissive than a typical enterprise network, meaning that we shouldn’t block all detected threats. Our goal was to keep attendees safe while also allowing them to learn about cybersecurity concepts and techniques. This included allowing demonstration of cyber-attack and defense techniques in controlled ways and permitting classes to train attendees where participants can practice new found skills in a sandbox environment. What isn’t tolerated, however, is for participants to use these new skills to attack each other or attack any infrastructure. If it’s illegal in the real world, it’s still illegal in the conference and must be shut down.

Different Skill Levels

The team investigating these threats included a mix of experienced and new analysts, for some, this was their first time in a SOC and first time using the full SOC tool flow. In the SOC we had a few rules:

1. Leave your ego at the door
2. Be curious, ask questions, and dig deep
3. Share your knowledge and experience; everyone is an expert at something.

We had a good mix of tier 1- 3 analysts and followed an escalation procedure where only some incidents were raised to the attention of the tier 3 analysts. Our goal was to handle as many incidents as possible with tier 1 and 2, allowing the tier 3 experts to spend more time on deep threat hunting, innovating, and automating the SOC.

We typically had only 1.5 days to set up the SOC operations, and less than a few hours to train everyone on the workflow and procedures. This emphasized the need for streamlined onboarding, integrated workflows, and automation where possible. Some of the Tier 1 analysts were able to identify, report, and block serious threats in their first few hours.

Sharing our learnings with others

Over the year we ran well over 100 tours of the SOC to share our learnings with others on all aspects of the SOC including People, Process and Technology used in the SOC, threats we have responded to, and security metrics that we gather.  These sessions have been interactive with great questions and feedback: the level of interest has been extremely high.

People are curious as to what we see on the network and how we go about protecting each event. We always have something interesting – and perhaps a little frightening – to share at each event.

Innovations and Improvements to the SOC

We use these learnings to evolve the SOC architecture to help us be much more effective at these events. Many of these improvements are developed and deployed live during SOC operations. Each time we get together, it’s like an intense hackathon where new capabilities are introduced while we operate. Below is a summary of the Endace contributions to SOC innovation. There were many more that the Cisco team also added.

1. Improved Capture Density: The first SOCs deployed 864TB of HDD storage in 8RU of rack space, which was overkill for these 7-day events. After Cisco Live USA, we retrofitted the SOC-in-a-Box with 244TB of NVMe storage in 2RU of rack space using 2 of our latest generation EndaceProbe 94C8-G5 models. Using two appliances gives us redundancy in case something fails, and provides up to 200Gbps capture bandwidth, way more than we need at these events.
2. Real Time File Extraction and Submission with Deduplication: Initially deployed at RSAC and evolved at each new event, real time file extraction uses Zeek hosted on EndaceProbe to extract any files from packet data and submit to an external sandbox such as Splunk Attack Analyser. We’ve improved it further with filtering, additional mime types, deduplication, and robust redundancy. Deduplication was the most recent innovation at Cisco Live APJC, which resulted in a dramatic reduction in the number of files submitted to Splunk Attack Analyzer (SAA). See Caleb Millar’s blog for more details.
3. Automating Mundane Tasks: We overwhelmed the Tier 1 analysts at Cisco Live USA with more password events than they could handle, so the team set out to automate. Now when credentials are detected in the clear, our automation will send an email to the affected account owner. This was a huge productivity boost to the whole SOC team who could now focus on more challenging threats and other automation tasks.
4. New Endace Vault API and XDR integration: This new API allows us to permanently archive important PCAP’s and provide them to XDR users in the Worklog of the incident. This allowed our Tier 1/2 analysts to make use of packet evidence without having to be an expert in the Endace GUI, with just one click analysts can view packet data to fully understand threats.
5. Dark Mode GUI: Every SOC analyst needs dark mode, and now it’s a feature of Endace!
6. Splunk Dashboard representing Endace: Delivered with at first RSAC which we have continued to refine and improve at every SOC event.
7. Endace SSO integration via DUO: At Cisco Live APJC we prototyped our Duo integration using SAML to provide users with SSO. This significantly reduces the time taken to onboard the SOC team, most of whom are new at every event.
8. Automated Deployment: We’ve scripted more of the setup to shorten the time it takes to get up and running. It now takes just an hour or two to have all the Endace capability running at any SOC event.

Open Architecture Makes it Possible

This rapid pace of innovation was only possible because of the open architecture of the Cisco products we integrated with, especially Splunk ES and Cisco XDR. These products allowed us to develop new dashboards and workflows without needing help from the Cisco team, we were able to experiment on our own and bring new capability that we could further tune at the SOC. The resultant architecture has proven itself extremely effective and these innovations will be published for commercial customers to adopt.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Bair Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with, and we came away with a great appreciation for the power of the Cisco Security tools. The Endace team was able to prove integration of innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Caleb Millar, Staff Software Engineer, Endace

---

Summary

At Cisco Live APJC 2025 Endace provided Full Packet Capture and real-time File Reconstruction from network traffic as part of the Security Operations Center (SOC).

File reconstruction recreates files from network packet data and submits each reconstructed file to Splunk Attack Analyzer (SAA) and Secure Malware Analytics (SMA) to detect threats. The submit rate approaches 10,000 file samples for the busiest day at a large event, even after filtering.

To reduce load on SAA & SMA, new support was added to detect duplicate files across multiple Zeek VMs. We implemented this using a Splunk app key value store to act as a centralized database of previously submitted files. The result was a 70% reduction in the number of files submitted to SAA, and this allowed us to still record the blast radius of all devices that handled a potentially malicious file.

Optimizing Analysis of Reconstructed Packet Data

EndaceProbe provides continuous packet capture and simultaneously hosts virtual machines to analyze network traffic in real time using commercial or open-source tools.

At previous events, we deployed the capability to reconstruct files from packet data using Zeek. Every file extraction is logged with details such as IP address, port, and mime type. All Zeek logs are sent to Splunk for indexing and searching, and the file itself is sent to Splunk Attack Analyzer (SAA) for analysis.

Over a typical week of SOC operation, we would reconstruct and submit up to 30,000 files to SAA, so we set ourselves a task to reduce the number by not submitting duplicates of files that we already submitted.

SAA is an automated threat analysis tool that detects threats in files like phishing or malware using a series of threat analysis tools. For example, an `.exe` will be detonated in a sandbox environment to check for malicious behavior. SAA then assigns a score to each submission as an indicator of risk and may submit some files for further analysis to Secure Malware Analytics (SMA). The results and risk score are recorded in Splunk and Cisco XDR.

EndaceProbe also has built in capability for file extraction, accessible via the GUI, that allows a user to retrospectively analyze a set of captured packets to reconstruct files and generate Zeek logs. This is useful for reviewing any files that may not have been submitted in real-time.

For Cisco Live Melbourne, continuous packet capture was handled by two EndaceProbes, each running three Zeek virtual machines, for a total of six virtual machine instances.

Due to the high number of files extracted from captured packet data (at Cisco Live Melbourne we extracted 370,000+ files) it is not practical to upload every single file to Splunk Attack Analyzer. For this reason, we prioritize higher risk files (such as .exe) and skip uploading high volume/low risk files (such a .pem extracted from each TLS connection).

Since all file extractions continue to be logged and indexed in Splunk, it is always possible to re-run a file extraction using EndaceProbe by searching against relevant IP addresses. This supports the use-case where deeper investigation is required on a file which has not previously been uploaded to Splunk Attack Analyzer.

De-Duplicating Submissions

During the conference we noticed a high number of duplicate files being submitted to Splunk Attack Analyzer. It is not unexpected to find duplicate files, especially requests for software updates and other services accessed by many users. However, submitting duplicate files does create unnecessary resource cost with diminishing value. It also potentially delays submission and analysis of new and interesting files as the list of uploads are queued to distribute load over time.

• Therefore, we started working on a solution to skip previously submitted files. A de-duplication solution should have some key features:
• Support de-duplication across multiple virtual machine instances.
• Not introduce significant overhead if new API requests are required.
• Blast radius mapping – searching for all instances of a file that was seen on the network. Importantly, if a malicious file is detected, it should be possible to search and build events which check for all past and future instances.

To address these requirements, we settled on using a key-value store (KV store) as a centralized database for all files uploaded to Splunk Attack Analyzer.

One option for a KV store implementation is provided via Splunk apps. This solution had some key benefits:

• Allows virtual machines running Zeek to be re-deployed, or new instances added, while retaining the same duplication database. In practice, this means duplicates can be detected across all virtual machines running Zeek.
• Simplified deployment: since Splunk is already running as part of the SOC, there is no need to provision additional storage and other resources.
• Creates new opportunities to integrate key-value data with Splunk searches and apps.

The “key” for our de-duplication case is the SHA256 checksum of the file. SHA256 was used to match the checksum Splunk Attack Analyzer included on the submission result page.

Technically, this is enough to meet our requirements, but the additional value field in the key-value store allows additional metadata to be included for a given file submission. The file extraction logs were also updated to include the SHA256 checksum. This allows all instances of a file to be searched from within Splunk.

The Results

Enabling de-duplication provided immediate benefit. With many extracted files detected as duplicates, as seen in the figure. The items shown in purple are files which would have previously been submitted but now can be skipped.
In total, 6513 files were skipped due to duplication, with a total of 14369 files submitted to Splunk Attack Analyzer during Cisco Live APJC.

This shows immediate cost savings and creates an improved list of focused Splunk Attack Analyzer submissions. Integrating the duplication index with Splunk KV Store also provides additional future opportunities to integrate with other Splunk data and create new tools while keeping deployment simple.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/