Category: Software

By Cary Wright, VP Product, Endace

Diverse Dataset over 5 Major Conferences

Over the five deployments the SOC architecture was subject to a variety of traffic in North America, Asia, and Australia, with attendees representing most regions. Some interesting stats from what we saw:

A Wide Variety of Threats

We’ve investigated and responded to a wide variety of threats, from simple passwords in the clear, to beaconing, RATs, port scanning, owned hosts, infected files, insecure applications, AI generated malicious domains, potential APTs obfuscating their C2 communications, exploits of known vulnerabilities and new novel threats.

There were also a bunch of false positives that we needed to run down. With Endace continuous packet capture integrated with the Cisco security stack we were able to dig deep to understand even the most challenging threats. By recording every packet from start of show to the very last moments we could arm the analysts with the evidence they needed to hunt down all manner of threats, if we were only capturing based on triggers or events we would have missed many of the threats that we did discover.

A great example of a threat we identified and responded to is captured by Daniel Lawson’s blog: Endace Full Packet Capture finds Active Directory Credentials in Clear Text.

For these cybersecurity conferences our environment needed to be more permissive than a typical enterprise network, meaning that we shouldn’t block all detected threats. Our goal was to keep attendees safe while also allowing them to learn about cybersecurity concepts and techniques. This included allowing demonstration of cyber-attack and defense techniques in controlled ways and permitting classes to train attendees where participants can practice new found skills in a sandbox environment. What isn’t tolerated, however, is for participants to use these new skills to attack each other or attack any infrastructure. If it’s illegal in the real world, it’s still illegal in the conference and must be shut down.

Different Skill Levels

The team investigating these threats included a mix of experienced and new analysts, for some, this was their first time in a SOC and first time using the full SOC tool flow. In the SOC we had a few rules:

1. Leave your ego at the door
2. Be curious, ask questions, and dig deep
3. Share your knowledge and experience; everyone is an expert at something.

We had a good mix of tier 1- 3 analysts and followed an escalation procedure where only some incidents were raised to the attention of the tier 3 analysts. Our goal was to handle as many incidents as possible with tier 1 and 2, allowing the tier 3 experts to spend more time on deep threat hunting, innovating, and automating the SOC.

We typically had only 1.5 days to set up the SOC operations, and less than a few hours to train everyone on the workflow and procedures. This emphasized the need for streamlined onboarding, integrated workflows, and automation where possible. Some of the Tier 1 analysts were able to identify, report, and block serious threats in their first few hours.

Sharing our learnings with others

Over the year we ran well over 100 tours of the SOC to share our learnings with others on all aspects of the SOC including People, Process and Technology used in the SOC, threats we have responded to, and security metrics that we gather.  These sessions have been interactive with great questions and feedback: the level of interest has been extremely high.

People are curious as to what we see on the network and how we go about protecting each event. We always have something interesting – and perhaps a little frightening – to share at each event.

Innovations and Improvements to the SOC

We use these learnings to evolve the SOC architecture to help us be much more effective at these events. Many of these improvements are developed and deployed live during SOC operations. Each time we get together, it’s like an intense hackathon where new capabilities are introduced while we operate. Below is a summary of the Endace contributions to SOC innovation. There were many more that the Cisco team also added.

1. Improved Capture Density: The first SOCs deployed 864TB of HDD storage in 8RU of rack space, which was overkill for these 7-day events. After Cisco Live USA, we retrofitted the SOC-in-a-Box with 244TB of NVMe storage in 2RU of rack space using 2 of our latest generation EndaceProbe 94C8-G5 models. Using two appliances gives us redundancy in case something fails, and provides up to 200Gbps capture bandwidth, way more than we need at these events.
2. Real Time File Extraction and Submission with Deduplication: Initially deployed at RSAC and evolved at each new event, real time file extraction uses Zeek hosted on EndaceProbe to extract any files from packet data and submit to an external sandbox such as Splunk Attack Analyser. We’ve improved it further with filtering, additional mime types, deduplication, and robust redundancy. Deduplication was the most recent innovation at Cisco Live APJC, which resulted in a dramatic reduction in the number of files submitted to Splunk Attack Analyzer (SAA). See Caleb Millar’s blog for more details.
3. Automating Mundane Tasks: We overwhelmed the Tier 1 analysts at Cisco Live USA with more password events than they could handle, so the team set out to automate. Now when credentials are detected in the clear, our automation will send an email to the affected account owner. This was a huge productivity boost to the whole SOC team who could now focus on more challenging threats and other automation tasks.
4. New Endace Vault API and XDR integration: This new API allows us to permanently archive important PCAP’s and provide them to XDR users in the Worklog of the incident. This allowed our Tier 1/2 analysts to make use of packet evidence without having to be an expert in the Endace GUI, with just one click analysts can view packet data to fully understand threats.
5. Dark Mode GUI: Every SOC analyst needs dark mode, and now it’s a feature of Endace!
6. Splunk Dashboard representing Endace: Delivered with at first RSAC which we have continued to refine and improve at every SOC event.
7. Endace SSO integration via DUO: At Cisco Live APJC we prototyped our Duo integration using SAML to provide users with SSO. This significantly reduces the time taken to onboard the SOC team, most of whom are new at every event.
8. Automated Deployment: We’ve scripted more of the setup to shorten the time it takes to get up and running. It now takes just an hour or two to have all the Endace capability running at any SOC event.

Open Architecture Makes it Possible

This rapid pace of innovation was only possible because of the open architecture of the Cisco products we integrated with, especially Splunk ES and Cisco XDR. These products allowed us to develop new dashboards and workflows without needing help from the Cisco team, we were able to experiment on our own and bring new capability that we could further tune at the SOC. The resultant architecture has proven itself extremely effective and these innovations will be published for commercial customers to adopt.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Bair Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with, and we came away with a great appreciation for the power of the Cisco Security tools. The Endace team was able to prove integration of innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Caleb Millar, Staff Software Engineer, Endace

Summary

At Cisco Live APJC 2025 Endace provided Full Packet Capture and real-time File Reconstruction from network traffic as part of the Security Operations Center (SOC).

File reconstruction recreates files from network packet data and submits each reconstructed file to Splunk Attack Analyzer (SAA) and Secure Malware Analytics (SMA) to detect threats. The submit rate approaches 10,000 file samples for the busiest day at a large event, even after filtering.

To reduce load on SAA & SMA, new support was added to detect duplicate files across multiple Zeek VMs. We implemented this using a Splunk app key value store to act as a centralized database of previously submitted files. The result was a 70% reduction in the number of files submitted to SAA, and this allowed us to still record the blast radius of all devices that handled a potentially malicious file.

Optimizing Analysis of Reconstructed Packet Data

EndaceProbe provides continuous packet capture and simultaneously hosts virtual machines to analyze network traffic in real time using commercial or open-source tools.

At previous events, we deployed the capability to reconstruct files from packet data using Zeek. Every file extraction is logged with details such as IP address, port, and mime type. All Zeek logs are sent to Splunk for indexing and searching, and the file itself is sent to Splunk Attack Analyzer (SAA) for analysis.

Over a typical week of SOC operation, we would reconstruct and submit up to 30,000 files to SAA, so we set ourselves a task to reduce the number by not submitting duplicates of files that we already submitted.

SAA is an automated threat analysis tool that detects threats in files like phishing or malware using a series of threat analysis tools. For example, an `.exe` will be detonated in a sandbox environment to check for malicious behavior. SAA then assigns a score to each submission as an indicator of risk and may submit some files for further analysis to Secure Malware Analytics (SMA). The results and risk score are recorded in Splunk and Cisco XDR.

EndaceProbe also has built in capability for file extraction, accessible via the GUI, that allows a user to retrospectively analyze a set of captured packets to reconstruct files and generate Zeek logs. This is useful for reviewing any files that may not have been submitted in real-time.

For Cisco Live Melbourne, continuous packet capture was handled by two EndaceProbes, each running three Zeek virtual machines, for a total of six virtual machine instances.

Due to the high number of files extracted from captured packet data (at Cisco Live Melbourne we extracted 370,000+ files) it is not practical to upload every single file to Splunk Attack Analyzer. For this reason, we prioritize higher risk files (such as .exe) and skip uploading high volume/low risk files (such a .pem extracted from each TLS connection).

Since all file extractions continue to be logged and indexed in Splunk, it is always possible to re-run a file extraction using EndaceProbe by searching against relevant IP addresses. This supports the use-case where deeper investigation is required on a file which has not previously been uploaded to Splunk Attack Analyzer.

De-Duplicating Submissions

During the conference we noticed a high number of duplicate files being submitted to Splunk Attack Analyzer. It is not unexpected to find duplicate files, especially requests for software updates and other services accessed by many users. However, submitting duplicate files does create unnecessary resource cost with diminishing value. It also potentially delays submission and analysis of new and interesting files as the list of uploads are queued to distribute load over time.

• Therefore, we started working on a solution to skip previously submitted files. A de-duplication solution should have some key features:
• Support de-duplication across multiple virtual machine instances.
• Not introduce significant overhead if new API requests are required.
• Blast radius mapping – searching for all instances of a file that was seen on the network. Importantly, if a malicious file is detected, it should be possible to search and build events which check for all past and future instances.

To address these requirements, we settled on using a key-value store (KV store) as a centralized database for all files uploaded to Splunk Attack Analyzer.

One option for a KV store implementation is provided via Splunk apps. This solution had some key benefits:

• Allows virtual machines running Zeek to be re-deployed, or new instances added, while retaining the same duplication database. In practice, this means duplicates can be detected across all virtual machines running Zeek.
• Simplified deployment: since Splunk is already running as part of the SOC, there is no need to provision additional storage and other resources.
• Creates new opportunities to integrate key-value data with Splunk searches and apps.

The “key” for our de-duplication case is the SHA256 checksum of the file. SHA256 was used to match the checksum Splunk Attack Analyzer included on the submission result page.

Technically, this is enough to meet our requirements, but the additional value field in the key-value store allows additional metadata to be included for a given file submission. The file extraction logs were also updated to include the SHA256 checksum. This allows all instances of a file to be searched from within Splunk.

The Results

Enabling de-duplication provided immediate benefit. With many extracted files detected as duplicates, as seen in the figure. The items shown in purple are files which would have previously been submitted but now can be skipped.
In total, 6513 files were skipped due to duplication, with a total of 14369 files submitted to Splunk Attack Analyzer during Cisco Live APJC.

This shows immediate cost savings and creates an improved list of focused Splunk Attack Analyzer submissions. Integrating the duplication index with Splunk KV Store also provides additional future opportunities to integrate with other Splunk data and create new tools while keeping deployment simple.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Peter Watt, Senior Sales and Partner Integration Engineer, Endace

Summary

At Cisco Live APJC 2025 the SOC has a robust procedure in place to identify cleartext passwords, and through use of automation with Splunk/XDR to notify users directly via-email of their potential exposure – offering them support through the SOC on-site, during the conference.

This is a powerful capability of the SOC. It does however, require a valid email address – which is found within SMTP, IMAP and POP3.
When cleartext Passwords, and other PII are identified outside email based protocols … it becomes more difficult.

What we Identified

Firstly, through automation, the SOC was able to identify the use of a cleartext password:

From within the incident in XDR, a https reference link is provided to provide direct access to the packets of interest, stored on the Endace Packet Capture Appliance.

Using the session construction capabilities of Wireshark – we were able to reconstruct the data stream, and identify an FTP session had taken place with the following actions:

This exposed:

• External IP Address
• Username,
• Password
• Directory
• Filename
• Internal IP address
• Filetype being transferred

Missing:

• Email Address

Reverse lookup of the IP address was able to shed further light upon the situation. But we still needed additional information. Which is where the full packet capture became valuable.

Using Endace’s recorded full packet capture data, we extracted packet data between the source and destination IP’s and found that the FTP was also taking place on a non-standard port:

Upon reconstructing the session, the file being transferred was able to be reassembled too. The file format can be clearly seen in the following:

Piecing together all the components above, we were able to identify the source of the FTP, notify and educate them to resolve a potential security threat.

Having an FTP server open and accessible on the internet with clear text passwords is very risky indeed. An attacker that obtains credentials could easily access and download any private or sensitive content, and more concerning they could place infected binaries or malware on the server with the intention to infect and own any machines accessing content on that server. The users were shocked to learn about this exposure and immediately stopped using the FTP service.

The same methodology was used during threat hunting where other cleartext PII was identified.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/ 

By Daniel Lawson, Senior Engineering Manager – Software, Endace
and Shaun Coulter, Technical Solutions Architect, Cisco

Summary

The Mission of the SOC at events like Cisco Live is:

Protect, Educate, Innovate.

While at Cisco Live APJC 2025, we were able to meet these objectives in a very direct way.

When Threat Hunting using full packet capture data we quickly identified a series of unencrypted Active Directory connections from conference attendees to their organization’s AD servers. Within an hour of identifying and documenting the issue the SOC had contacted the organization’s IT team and invited team members who were attending the conference to the SOC to walk them through the evidence.

We were only able to discover this serious security issue because EndaceProbe had recorded full continuous packet data for the entire conference, and EndaceVision provided us with a UI to rapidly threat hunt by searching, visualizing, and analyzing the packet data.

Curious Threat Hunt Leads to Rapid Discovery

During one of the SOC tours, one of my fellow engineers was discussing having identified some Kerberos traffic on the network. This prompted me to wonder if there was any LDAP traffic, as that is another common authentication protocol, and while it should be deployed with an encrypted communication channel, it isn’t always.

Sure enough, an EndaceVision query with an “Application is ldap” filter showed some unencrypted LDAP traffic!

Sending this traffic to Endace’s hosted Wireshark allowed me to view the raw packet decodes and confirm that these were completed connections, involving what looked like a service account and a password.

The device went on to request some entries from the organizations AD Global Address List, returning email addresses, job titles, and phone numbers. The GAL may also include other job and address information, and photos. Had an attacker found this information, at a minimum this password leakage could have resulted in the exfiltration of the organization’s complete address book, which may have been used for spamming or identify theft.

Depending on the nature of the service account that was leaked however, this might have resulted in a wider compromise: it might also have been a valid login account and therefore a foothold in the organization’s network, and potentially further scope for lateral movement within the network.

We used a Conversations Chords Chart to see if this involved other attendees, this showed us four local devices communicating with three LDAP servers for that organization.

The SOC was able to contact the organization involved and get in touch with the attendees, and they were invited down to the SOC where they got a detailed one-on-one demonstration of how we found the insecurity using the capabilities of Cisco and Endace SOC and the benefits of full packet capture.

The organization was both surprised and concerned with the data that had been exposed and stated that they would address the finding immediately.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/