What We’ve Learned After Five Incredible SOC Events

By Cary Wright, VP Product, Endace

---

Diverse Dataset over 5 Major Conferences

Over the five deployments the SOC architecture was subject to a variety of traffic in North America, Asia, and Australia, with attendees representing most regions. Some interesting stats from what we saw:

Attendees	109,500 (over 5 conferences)
Packets Captured (TB)	204.8 Terabytes (236 Billion packets)
Unique Hosts	129,021
Sessions	2.775 (Billion)
Files Extracted by Endace	1,461,000
Files submitted to Splunk Attack Analyzer	86,000
Files submitted to Secure Malware Analytics	24,700
Password in the clear events	9,527
Devices with Password in the clear	291
Logs sent to Splunk (M)	6.75 Billion
DNS requests	428 Million
Encrypted traffic	82%

A Wide Variety of Threats

We’ve investigated and responded to a wide variety of threats, from simple passwords in the clear, to beaconing, RATs, port scanning, owned hosts, infected files, insecure applications, AI generated malicious domains, potential APTs obfuscating their C2 communications, exploits of known vulnerabilities and new novel threats.

There were also a bunch of false positives that we needed to run down. With Endace continuous packet capture integrated with the Cisco security stack we were able to dig deep to understand even the most challenging threats. By recording every packet from start of show to the very last moments we could arm the analysts with the evidence they needed to hunt down all manner of threats, if we were only capturing based on triggers or events we would have missed many of the threats that we did discover.

A great example of a threat we identified and responded to is captured by Daniel Lawson’s blog: Endace Full Packet Capture finds Active Directory Credentials in Clear Text.

For these cybersecurity conferences our environment needed to be more permissive than a typical enterprise network, meaning that we shouldn’t block all detected threats. Our goal was to keep attendees safe while also allowing them to learn about cybersecurity concepts and techniques. This included allowing demonstration of cyber-attack and defense techniques in controlled ways and permitting classes to train attendees where participants can practice new found skills in a sandbox environment. What isn’t tolerated, however, is for participants to use these new skills to attack each other or attack any infrastructure. If it’s illegal in the real world, it’s still illegal in the conference and must be shut down.

Different Skill Levels

The team investigating these threats included a mix of experienced and new analysts, for some, this was their first time in a SOC and first time using the full SOC tool flow. In the SOC we had a few rules:

1. Leave your ego at the door
2. Be curious, ask questions, and dig deep
3. Share your knowledge and experience; everyone is an expert at something.

We had a good mix of tier 1- 3 analysts and followed an escalation procedure where only some incidents were raised to the attention of the tier 3 analysts. Our goal was to handle as many incidents as possible with tier 1 and 2, allowing the tier 3 experts to spend more time on deep threat hunting, innovating, and automating the SOC.

We typically had only 1.5 days to set up the SOC operations, and less than a few hours to train everyone on the workflow and procedures. This emphasized the need for streamlined onboarding, integrated workflows, and automation where possible. Some of the Tier 1 analysts were able to identify, report, and block serious threats in their first few hours.

Sharing our learnings with others

Over the year we ran well over 100 tours of the SOC to share our learnings with others on all aspects of the SOC including People, Process and Technology used in the SOC, threats we have responded to, and security metrics that we gather.  These sessions have been interactive with great questions and feedback: the level of interest has been extremely high.

People are curious as to what we see on the network and how we go about protecting each event. We always have something interesting – and perhaps a little frightening – to share at each event.

Innovations and Improvements to the SOC

We use these learnings to evolve the SOC architecture to help us be much more effective at these events. Many of these improvements are developed and deployed live during SOC operations. Each time we get together, it’s like an intense hackathon where new capabilities are introduced while we operate. Below is a summary of the Endace contributions to SOC innovation. There were many more that the Cisco team also added.

1. Improved Capture Density: The first SOCs deployed 864TB of HDD storage in 8RU of rack space, which was overkill for these 7-day events. After Cisco Live USA, we retrofitted the SOC-in-a-Box with 244TB of NVMe storage in 2RU of rack space using 2 of our latest generation EndaceProbe 94C8-G5 models. Using two appliances gives us redundancy in case something fails, and provides up to 200Gbps capture bandwidth, way more than we need at these events.
2. Real Time File Extraction and Submission with Deduplication: Initially deployed at RSAC and evolved at each new event, real time file extraction uses Zeek hosted on EndaceProbe to extract any files from packet data and submit to an external sandbox such as Splunk Attack Analyser. We’ve improved it further with filtering, additional mime types, deduplication, and robust redundancy. Deduplication was the most recent innovation at Cisco Live APJC, which resulted in a dramatic reduction in the number of files submitted to Splunk Attack Analyzer (SAA). See Caleb Millar’s blog for more details.
3. Automating Mundane Tasks: We overwhelmed the Tier 1 analysts at Cisco Live USA with more password events than they could handle, so the team set out to automate. Now when credentials are detected in the clear, our automation will send an email to the affected account owner. This was a huge productivity boost to the whole SOC team who could now focus on more challenging threats and other automation tasks.
4. New Endace Vault API and XDR integration: This new API allows us to permanently archive important PCAP’s and provide them to XDR users in the Worklog of the incident. This allowed our Tier 1/2 analysts to make use of packet evidence without having to be an expert in the Endace GUI, with just one click analysts can view packet data to fully understand threats.
5. Dark Mode GUI: Every SOC analyst needs dark mode, and now it’s a feature of Endace!
6. Splunk Dashboard representing Endace: Delivered with at first RSAC which we have continued to refine and improve at every SOC event.
7. Endace SSO integration via DUO: At Cisco Live APJC we prototyped our Duo integration using SAML to provide users with SSO. This significantly reduces the time taken to onboard the SOC team, most of whom are new at every event.
8. Automated Deployment: We’ve scripted more of the setup to shorten the time it takes to get up and running. It now takes just an hour or two to have all the Endace capability running at any SOC event.

Open Architecture Makes it Possible

This rapid pace of innovation was only possible because of the open architecture of the Cisco products we integrated with, especially Splunk ES and Cisco XDR. These products allowed us to develop new dashboards and workflows without needing help from the Cisco team, we were able to experiment on our own and bring new capability that we could further tune at the SOC. The resultant architecture has proven itself extremely effective and these innovations will be published for commercial customers to adopt.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Bair Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with, and we came away with a great appreciation for the power of the Cisco Security tools. The Endace team was able to prove integration of innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/