Uncategorized – Page 2

Cisco Live APJ 2025: Optimizing Analysis of Reconstructed Packet Data

Original Entry by : Caleb Millar

By Caleb Millar, Staff Software Engineer, Endace

Summary

At Cisco Live APJC 2025 Endace provided Full Packet Capture and real-time File Reconstruction from network traffic as part of the Security Operations Center (SOC).

File reconstruction recreates files from network packet data and submits each reconstructed file to Splunk Attack Analyzer (SAA) and Secure Malware Analytics (SMA) to detect threats. The submit rate approaches 10,000 file samples for the busiest day at a large event, even after filtering.

To reduce load on SAA & SMA, new support was added to detect duplicate files across multiple Zeek VMs. We implemented this using a Splunk app key value store to act as a centralized database of previously submitted files. The result was a 70% reduction in the number of files submitted to SAA, and this allowed us to still record the blast radius of all devices that handled a potentially malicious file.

Optimizing Analysis of Reconstructed Packet Data

EndaceProbe provides continuous packet capture and simultaneously hosts virtual machines to analyze network traffic in real time using commercial or open-source tools.

At previous events, we deployed the capability to reconstruct files from packet data using Zeek. Every file extraction is logged with details such as IP address, port, and mime type. All Zeek logs are sent to Splunk for indexing and searching, and the file itself is sent to Splunk Attack Analyzer (SAA) for analysis.

Over a typical week of SOC operation, we would reconstruct and submit up to 30,000 files to SAA, so we set ourselves a task to reduce the number by not submitting duplicates of files that we already submitted.

SAA is an automated threat analysis tool that detects threats in files like phishing or malware using a series of threat analysis tools. For example, an `.exe` will be detonated in a sandbox environment to check for malicious behavior. SAA then assigns a score to each submission as an indicator of risk and may submit some files for further analysis to Secure Malware Analytics (SMA). The results and risk score are recorded in Splunk and Cisco XDR.

EndaceProbe also has built in capability for file extraction, accessible via the GUI, that allows a user to retrospectively analyze a set of captured packets to reconstruct files and generate Zeek logs. This is useful for reviewing any files that may not have been submitted in real-time.

Submitted File list in Splunk Attack Analyzer — Submitted File List in Splunk Attack Analyzer

For Cisco Live Melbourne, continuous packet capture was handled by two EndaceProbes, each running three Zeek virtual machines, for a total of six virtual machine instances.

Due to the high number of files extracted from captured packet data (at Cisco Live Melbourne we extracted 370,000+ files) it is not practical to upload every single file to Splunk Attack Analyzer. For this reason, we prioritize higher risk files (such as .exe) and skip uploading high volume/low risk files (such a .pem extracted from each TLS connection).

Since all file extractions continue to be logged and indexed in Splunk, it is always possible to re-run a file extraction using EndaceProbe by searching against relevant IP addresses. This supports the use-case where deeper investigation is required on a file which has not previously been uploaded to Splunk Attack Analyzer.

De-Duplicating Submissions

During the conference we noticed a high number of duplicate files being submitted to Splunk Attack Analyzer. It is not unexpected to find duplicate files, especially requests for software updates and other services accessed by many users. However, submitting duplicate files does create unnecessary resource cost with diminishing value. It also potentially delays submission and analysis of new and interesting files as the list of uploads are queued to distribute load over time.

Therefore, we started working on a solution to skip previously submitted files. A de-duplication solution should have some key features:
Support de-duplication across multiple virtual machine instances.
Not introduce significant overhead if new API requests are required.
Blast radius mapping – searching for all instances of a file that was seen on the network. Importantly, if a malicious file is detected, it should be possible to search and build events which check for all past and future instances.

To address these requirements, we settled on using a key-value store (KV store) as a centralized database for all files uploaded to Splunk Attack Analyzer.

One option for a KV store implementation is provided via Splunk apps. This solution had some key benefits:

Allows virtual machines running Zeek to be re-deployed, or new instances added, while retaining the same duplication database. In practice, this means duplicates can be detected across all virtual machines running Zeek.
Simplified deployment: since Splunk is already running as part of the SOC, there is no need to provision additional storage and other resources.
Creates new opportunities to integrate key-value data with Splunk searches and apps.

The “key” for our de-duplication case is the SHA256 checksum of the file. SHA256 was used to match the checksum Splunk Attack Analyzer included on the submission result page.

Technically, this is enough to meet our requirements, but the additional value field in the key-value store allows additional metadata to be included for a given file submission. The file extraction logs were also updated to include the SHA256 checksum. This allows all instances of a file to be searched from within Splunk.

The Results

Results showing before and after de-duplication of file submissions was enabled.

Enabling de-duplication provided immediate benefit. With many extracted files detected as duplicates, as seen in the figure. The items shown in purple are files which would have previously been submitted but now can be skipped.
In total, 6513 files were skipped due to duplication, with a total of 14369 files submitted to Splunk Attack Analyzer during Cisco Live APJC.

This shows immediate cost savings and creates an improved list of focused Splunk Attack Analyzer submissions. Integrating the duplication index with Splunk KV Store also provides additional future opportunities to integrate with other Splunk data and create new tools while keeping deployment simple.

Acknowledgements

Once again, our thanks go to the Cisco team led by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the Cisco Live APJC SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools.

The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/

Cisco Live APJ 2025: Cleartext Passwords and Always on Packet Capture

Original Entry by : Peter Watt

By Peter Watt, Senior Sales and Partner Integration Engineer, Endace

Summary

At Cisco Live APJC 2025 the SOC has a robust procedure in place to identify cleartext passwords, and through use of automation with Splunk/XDR to notify users directly via-email of their potential exposure – offering them support through the SOC on-site, during the conference.

This is a powerful capability of the SOC. It does however, require a valid email address – which is found within SMTP, IMAP and POP3.
When cleartext Passwords, and other PII are identified outside email based protocols … it becomes more difficult.

What we Identified

Firstly, through automation, the SOC was able to identify the use of a cleartext password:

Cleartext Password Notification from Cisco XDR

From within the incident in XDR, a https reference link is provided to provide direct access to the packets of interest, stored on the Endace Packet Capture Appliance.

Using the session construction capabilities of Wireshark – we were able to reconstruct the data stream, and identify an FTP session had taken place with the following actions:

Reconstructed FTP Session in Wireshark (hosted on EndaceProbe)

This exposed:

External IP Address
Username,
Password
Directory
Filename
Internal IP address
Filetype being transferred

Missing:

Email Address

Reverse lookup of the IP address was able to shed further light upon the situation. But we still needed additional information. Which is where the full packet capture became valuable.

Using Endace’s recorded full packet capture data, we extracted packet data between the source and destination IP’s and found that the FTP was also taking place on a non-standard port:

FTP Session Running on a Non-Standard Port

Upon reconstructing the session, the file being transferred was able to be reassembled too. The file format can be clearly seen in the following:

File type is Identified from the Recorded Full Packet Capture Data

Piecing together all the components above, we were able to identify the source of the FTP, notify and educate them to resolve a potential security threat.

Having an FTP server open and accessible on the internet with clear text passwords is very risky indeed. An attacker that obtains credentials could easily access and download any private or sensitive content, and more concerning they could place infected binaries or malware on the server with the intention to infect and own any machines accessing content on that server. The users were shocked to learn about this exposure and immediately stopped using the FTP service.

The same methodology was used during threat hunting where other cleartext PII was identified.

Acknowledgements

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/

Cisco Live APJ 2025: Endace Full Packet Capture Finds Active Directory Credentials in Clear Text

Original Entry by : Daniel Lawson

By Daniel Lawson, Senior Engineering Manager – Software, Endace
and Shaun Coulter, Technical Solutions Architect, Cisco

Summary

The Mission of the SOC at events like Cisco Live is:

Protect, Educate, Innovate.

While at Cisco Live APJC 2025, we were able to meet these objectives in a very direct way.

When Threat Hunting using full packet capture data we quickly identified a series of unencrypted Active Directory connections from conference attendees to their organization’s AD servers. Within an hour of identifying and documenting the issue the SOC had contacted the organization’s IT team and invited team members who were attending the conference to the SOC to walk them through the evidence.

We were only able to discover this serious security issue because EndaceProbe had recorded full continuous packet data for the entire conference, and EndaceVision provided us with a UI to rapidly threat hunt by searching, visualizing, and analyzing the packet data.

Curious Threat Hunt Leads to Rapid Discovery

During one of the SOC tours, one of my fellow engineers was discussing having identified some Kerberos traffic on the network. This prompted me to wonder if there was any LDAP traffic, as that is another common authentication protocol, and while it should be deployed with an encrypted communication channel, it isn’t always.

Sure enough, an EndaceVision query with an “Application is ldap” filter showed some unencrypted LDAP traffic!

EndaceVision Query Shows Unencrypted LDAP traffic

Sending this traffic to Endace’s hosted Wireshark allowed me to view the raw packet decodes and confirm that these were completed connections, involving what looked like a service account and a password.

Analyzing Unencrypted LDAP traffic in Wireshark

The device went on to request some entries from the organizations AD Global Address List, returning email addresses, job titles, and phone numbers. The GAL may also include other job and address information, and photos. Had an attacker found this information, at a minimum this password leakage could have resulted in the exfiltration of the organization’s complete address book, which may have been used for spamming or identify theft.

Depending on the nature of the service account that was leaked however, this might have resulted in a wider compromise: it might also have been a valid login account and therefore a foothold in the organization’s network, and potentially further scope for lateral movement within the network.

We used a Conversations Chords Chart to see if this involved other attendees, this showed us four local devices communicating with three LDAP servers for that organization.

Viewing conversations in the EndaceVision chord chart

The SOC was able to contact the organization involved and get in touch with the attendees, and they were invited down to the SOC where they got a detailed one-on-one demonstration of how we found the insecurity using the capabilities of Cisco and Endace SOC and the benefits of full packet capture.

The organization was both surprised and concerned with the data that had been exposed and stated that they would address the finding immediately.

Members of the Cisco Live APJ 2025 SOC Team

Acknowledgements

Read related Cisco Team Blogs from the Cisco Live APJC SOC: https://blogs.cisco.com/security/cisco-live-melbourne-2025-soc

For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/

Govware 2025: Full packet capture reveals suspicious traffic masquerading as MPEG in HTTP POSTS

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace

At GovWare 2025, Suspicious Video traffic was investigated using Full Packet Capture, revealing the traffic was in fact not video at all, but instead a potential threat masquerading as video to evade detection. Without packet capture this traffic would have gone unnoticed.

In this blog we reveal what led us to be suspicious, how we analyzed the traffic, and the clues that lead us to believe it may be threat traffic to a concerning destination.

Endace Always-on Packet Capture

Two of Endace’s newest EP94C8 EndaceProbes were installed in the GovWare SOC to provide full packet capture to support the conference SOC directives of Protect, Educate, and Innovate. Full packet capture provides unique insight into all activities on the network, delivering critical context and evidence for Incident Response and Threat Hunting teams.

The EndaceProbes each hosted three VMs that were running Zeek and delivering critical log data into Splunk. Custom Zeek script additions provided additional valuable details around clear text passwords in email and HTTP, and the use of insecure protocols. Zeek was also used for file-carving and automated submission of objects to Splunk Attack Analyzer and a beta of Endace’s new Vault API used in a Cisco XDR automated workflow was field tested.

SOC Findings and Lessons Learned

During a cursory review of the file mime-types submitted to Splunk Attack Analyzer, a number of video files were observed. To ascertain the contents of these files, an interactive session was launched in Splunk Attack Analyzer and VLC used to open them. Nothing was observed in these media files, and no codec information was available, which lead to the question “Why did Zeek carve these files in the first place?”

A custom search in Splunk to identify more information about files with mime-type video/mpeg or video/mpv showed a curious pattern emerging. A number of these files were approximately 500 bytes in length – far too small to be a legitimate media file. All these small files were associated with various conference WIFI addresses and all being sent to the same remote IP address.

A bespoke investigation into these IP addresses in Cisco XDR indicated that there were connections to known and suspected malicious sites:

Using EndaceVision the relationship between these addresses could also be seen in the Chords chart where 5 infected hosts were communicating with a single external address. The thicker lines indicated larger data transfers:

Pivoting to the hosted Wireshark instance available on all EndaceProbes quickly revealed the data flows from the conference WIFI to the remote address to be in the form of HTTP POSTs:

Following the HTTP streams in Wireshark then revealed an interesting User-Agent:

A search of the web indicated that these Host and User-Agent strings were used by a now-defunct windows application called SIM Instant Messenger. The domain associated with this application was also defunct and indications are that the project was discontinued in 2008.

On the face of it, this could be an old instant messaging client contacting a central server, but the age of the application and the fact the contents were impossibly small MPEG files continued to raise suspicion. Information relevant to the destination IP address also raised doubts that this was legitimate messaging traffic, even within the context of being embedded inside another application.

The question about how these MPEG files were carved was soon answered by Wireshark:

The first eight bytes of each of these POSTs were associated with different MPEG magic numbers, a few of which are shown below:

Notice that the bytes following the magic numbers in the screenshot above are the same, and there was a lot of repetition of content across all the traffic to the external address – could this be compromised endpoints obfuscating their phone home connections to a C2? The short, directional and periodic nature of these connections could also be seen to support this theory.

Unfortunately, no further context was able to be extracted from this small sample of connections, but further monitoring of small POSTs with MPEG contents external sites would be warranted on a corporate network – we will certainly be looking for similar patterns at future events.

It’s important to note that this threat investigation could not have been conducted without continuous full packet capture provided by EndaceProbe. At no point was this traffic flagged or alerted on, therefore a triggered PCAP or metadata only solution would not have captured these packets and their presence on the network and the risk associated with them would have gone unseen by the network operators. When securing the most important networks in the world it’s critical to have full packet capture for visibility to defend against attackers that will go to extreme lengths to obfuscate their activity.

Acknowledgements

Once again, our thanks go to the Cisco team lead by @Jessica Oppenheimer for the opportunity to include EndaceProbes in the GovWare SOC architecture. The SOC team is a collection of Cisco experts across many domains who were a pleasure to work and innovate with and we came away with a great appreciation for power of the Cisco Security tools. The Endace team was able to prove out integration innovations from previous SOC events and test these in earnest in a real-world environment in preparation for making them generally available to the market.

About GovWare

GovWare Conference and Exhibition is the region’s premier cyber information and connectivity platform, offering multi-channel touchpoints to drive community intel sharing, training, and strategic collaborations.

A trusted nexus for over three decades, GovWare unites policymakers, tech innovators, and end-users across Asia and beyond, driving pertinent dialogues on the latest trends and critical information flow. It empowers growth and innovation through collective insights and partnerships.

Its success lies in the trust and support from the cybersecurity and broader cyber community that it has had the privilege to serve over the years, as well as organisational partners who share the same values and mission to enrich the cyber ecosystem.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Threat actors are recording PCAPs, maybe you should too?

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace

On Aug 29th, Government Cybersecurity agencies from around the world released a joint advisory detailing how nation-state threat actors are compromising networks across the world, particularly in the US, Australia, Canada, New Zealand and the UK: www.cisa.gov/news-events/cybersecurity-advisories/aa25-239a www.darkreading.com/cybersecurity-operations/cisa-fbi-nsa-warn-chinese-global-espionage-system

These attacks are primarily focussed on telecommunications, government, lodging and military networks, and the tactics, techniques, and procedures (TTP) overlaps with APT (Advanced Persistent Threat) actors linked to multiple China-based entities. These threat actors are exploiting well-known vulnerabilities in VPN servers and web user interfaces on switches and routers. Even devices not owned by targets of interest are being compromised in order to provide additional attack pathways to the intended targets. Upon gaining a foothold in a network, persistence is achieved by modifying ACLs, opening services on non-standard ports to avoid detection, and tunnelling C2 and exfiltrated data to obfuscate malicious activity.

Of note in this particular instance is the use of PCAP collection on the target network by the threat actors. Once they’ve gained a foothold on the network infrastructure, the native capability of some routers to record PCAPs is then used to capture TACACS+ (authentication) traffic. When transmitted in clear text (or weakly encrypted) this authentication traffic exposes users credentials which can then be used to elevate the attacker’s access and enable them to move laterally across the network.

The use of network sniffing to extract credentials in authentication traffic is a common technique of threat actors (attack.mitre.org/versions/v17/techniques/T1040/). As we continue to see the stubborn use of unencrypted and weakly-encrypted protocols on networks, these insecure communications remain prime targets for credential gathering. Additionally, the uses of maliciously collected PCAP is evolving, with the ArcaneDoor campaign taking this a step further and exfiltrating captured PCAPs for remote analysis (attack.mitre.org/campaigns/C0046/). Exfiltrated PCAPs may contain anything from authentication data to file objects.

PCAP data is the ground-truth for what is happening on the network, and it is the source that all other network and security telemetry is derived from. Threat actors know this and value the raw unfiltered and unsampled intel that it provides about the target. This begs the question: if your adversaries see value in collecting PCAPs off your network, shouldn’t you be capturing full PCAP too!?

If you are not recording your network traffic, your security team has less visibility into network activity than your attackers – which makes the job of protecting your network impossibly difficult. With PCAP at their fingertips, SOC analysts can see exactly what’s happening on the network, making for faster, more accurate investigation and resolution of security incidents – as this excellent blog post from Cisco’s Steve Nowell describes.

That full PCAP data is so valuable to attackers also highlights a stark warning that wise defenders should heed. PCAP data must be protected and secured to the highest standards. Can you trust packet capture solutions that aren’t FIPS and Common Criteria certified? Or packet capture sources that can’t be properly locked down and protected from access by attackers?

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Network Visibility in Action: Endace and Cisco Drive SOC Defenses at RSAC 2025

Original Entry by : Cary Wright

By Cary Wright, VP Product Management, Endace

Uncovering insights from the 6th Annual Security Operations Center at RSA Conference

For the sixth consecutive year, a dedicated Security Operations Center (SOC) monitored the RSA Conference (RSAC) network, protecting a dynamic environment serving over 40,000 attendees. A collaboration between Endace and Cisco (and other security partners), the SOC provided real-world insights into current threat landscapes and security challenges, and demonstrated the critical importance of comprehensive network monitoring and real-time threat detection in large-scale environments.

The 2025 SOC team consisted of:

5 Endace analysts
9 Cisco/Splunk analysts
3 dedicated threat hunters
3 managers

Network Monitoring at Unprecedented Scale

The SOC captured and analyzed an astounding volume of data flowing through the conference network:

40+ billion packets captured (more than double the 19 billion from the previous year)
33 TB of packet data (up from 17TB)
Peak bandwidth usage of 3.4 Gbps (up from 2.2 Gbps)
615 million total sessions (increased from 383 million)
793 million logs captured
287,000 files extracted with 26,374 submitted for deeper analysis

Endace’s VP of Product Management Cary Wright explained the scope: “We tapped into the network and recorded everything—all the packets that traveled across that network—approximately 30 terabytes of data over the course of the whole conference.”

The Technical Architecture: Integration in Action

The SOC implemented a sophisticated, multi-layered security architecture centered around visibility and integration:

1. Network Capture Layer: EndaceProbe appliances performed full packet capture, creating a complete record of all network activity.
2. Log Generation and Analysis: The Endace systems generated metadata through tools like Zeek, which was then forwarded to Splunk and Cisco security tools for analysis.
3. Threat Detection Systems: Cisco Secure Firewall provided intrusion detection (running in non-blocking mode to avoid disrupting vendor demonstrations while still identifying potential threats).
4. Integration Layer: All components were interconnected, allowing analysts to pivot seamlessly from alerts directly to the relevant packet data, providing context for rapid investigation.
5. File Analysis Pipeline: Files transmitted across the network were extracted and analyzed:

1. - 287,000+ files extracted from network traffic
  - 26,374 files sent to Splunk Attack Analyzer
  - 7,546 files forwarded to Cisco Malware Analytics for in-depth examination

Key Security Findings and Trends

The SOC’s monitoring revealed several concerning security trends:

1. Declining Encryption Levels

One surprising finding was a drop in the percentage of encrypted traffic, from approximately 80% in 2024 to 74% in 2025. This regression toward “the dark past” of unencrypted communications creates significant security vulnerabilities.

More troubling was the increase in weak encryption (TLS 1.0/1.1) to 40% of encrypted traffic, along with the continued presence of plaintext password transmission.

2. Plaintext Passwords Continue

Though trending downward over the years, plaintext passwords remain a persistent problem, showing that the power of a strong password is nothing without an encrypted communication protocol!

1. - 2020: 96,361 cleartext passwords (2,178 unique accounts)
  - 2022: 55,525 cleartext passwords (2,210 unique accounts)
  - 2023: 36,910 cleartext passwords (424 unique accounts)
  - 2024: 20,916 cleartext passwords (99 unique accounts)
  - 2025: 1,807 cleartext passwords (87 unique accounts)

3. Legacy Protocol Persistence: POP3 Refuses to Die

The SOC discovered continued use of vulnerable legacy protocols:

1. - POP3 (unencrypted email retrieval)
  - Non-secured SMTP (email transmission)
  - Unencrypted IMAP

4. Advanced Threat Techniques

The SOC identified several sophisticated attack techniques, including:

1. - New domain generation algorithm (DGA) approaches using combinations of 2-3 random words
  - Command and control (C2) traffic
  - Cleartext transmission of sensitive data
  - Unsecured translation services transmitting text and audio in the clear
  - Exposed CCTV camera feeds

The Value of Complete Network Visibility

The collaborative SOC deployment at RSAC 2025 demonstrated the crucial role that full packet capture plays in modern security operations. By capturing and analyzing every packet traversing the network, security teams gained:

1. - Complete visibility into all network communications
  - Contextual evidence for security investigations
  - Rapid response capabilities through integrated tools
  - Retrospective analysis of historical network data

The integration between Endace’s packet capture technology and Cisco’s security suite enabled a powerful workflow: alerts from security tools could be immediately investigated by pivoting directly to the relevant network traffic, dramatically reducing investigation time.

Key Takeaways for Security Teams

Based on the RSAC 2025 SOC experience, organizations should consider these best practices:

1. - Deploy comprehensive network monitoring with full packet capture for complete visibility
  - Implement integrated security tools that work together seamlessly
  - Focus on encryption enforcement to protect sensitive data in transit
  - Eliminate legacy protocols that transmit data in cleartext
  - Use personal VPNs when connecting to public networks
  - Keep operating systems patched and maintain robust configuration management

The Endace and Cisco-powered SOC at RSAC 2025 demonstrated that comprehensive network visibility remains fundamental to effective security operations. As threats grow more sophisticated, the ability to see, analyze, and respond to every packet traversing the network becomes increasingly critical.

By integrating full packet capture with advanced security analytics, organizations can build security operations centers that provide both the breadth and depth of visibility needed to detect and respond to today’s most sophisticated threats.

This blog post is based on information shared during the “PROTECTED: The 6th Annual Report from the SOC at RSAC” session at RSA Conference 2025.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Cisco Live US 2025 SOC – PCAP or it didn’t happen!

Original Entry by : Barry "Baz" Shaw

By Barry “Baz” Shaw, Senior Engineering Manager, Technology Partner Programs, Endace

Elevating Incident Response with the Ultimate Network Forensics

After a successful SOC @ RSAC2025, our team was stoked to be invited to help Cisco run the SOC at Cisco Live US (CLUS). We jumped at the chance to work with the Cisco team again. It was a great opportunity to innovate while helping Cisco protect and educate the attendees of the conference. Plus, the Cisco team is a lot of fun to hang out with—there’s a very infectious vibe in the SOC that has everyone buzzing for the entire week.

Packet capture is essential in the SOC. It provides an indelible record of all network activity, which is invaluable to the SOC team when investigating threats or security risks—hence the phrase, “PCAP or it didn’t happen.” For CLUS, we deployed two EndaceProbes with a combined storage of 864TB to continuously record all network activity delivered via 2 x 10GbE SPAN ports. This gave us the capacity to record at least several weeks of full network packet data—covering more than the entire duration of the show.

Endace Fusion integrations provided the glue between the Cisco Security suite and the packet data on the EndaceProbes, enabling analysts to quickly pivot from Splunk Enterprise Security/Splunk Cloud, Cisco XDR, Cisco Firepower, and Cisco Secure Network Analytics (SNA) through to EndaceVision and hosted Wireshark. When access to historical full PCAP is available in a seamless integration, security analysts are empowered with the ease of contextual access, and this simplifies the use of PCAP data, where previously it could be seen to be cumbersome to use.

The EndaceProbes each hosted two VMs that were running Zeek and delivering critical log data into Splunk. Custom Zeek script additions provided additional valuable detail around clear text passwords in email and HTTP, shining a light on the heavy use of insecure protocols, and ultimately driving automation to manage the unexpected volume. File-carving was enabled, and over 750,000 files were reconstructed from packet data, with over 40,000 samples submitted to Splunk Attack Analyzer (SAA) via Endace’s automatic submission software. SAA then sent over 12,000 files to Secure Malware Analytics (formerly Threat Grid) for dynamic analysis of the behavior.

SOC Findings and Lessons Learned

The SOC team was surprised and initially overwhelmed at the volume of unencrypted traffic on the network. Logging of passwords was coupled with a Cisco XDR automation that created an incident on each detection. This resulted in a heavy workload identifying and notifying users to educate and protect them in the future. The Splunk team developed a creative automated solution to notify users that the SOC detected their use of insecure protocols.

We even found a version of POP that was news to us all—APOP. This hashes the server timestamp in the response header with the user’s password to create a password digest. While this obscures the password, it only delays its inevitable retrieval, all the while the actual message bodies are still transferred in plain text!

In the theme of plain text passwords, reviewing the connections associated with one of these sessions showed a large number of file downloads in the Zeek log generated on EndaceProbe. This was one of many clients that used the free conference Wi-Fi to download Windows update files, but after filtering out the cab files in a Splunk search, we found a suspicious-looking file:

A search on this filename in SAA confirmed the presence of a malware download by this unfortunate user, whom the SOC team made every effort to identify.

There were also a few notable occasions where Secure Firewall alerts indicated intrusion attempts, which, after a pivot to EndaceVision, were pulled up in Wireshark for further analysis. One alert of note was a “BROWSER-IE Microsoft Internet Explorer Chakra.dll Array.filter type confusion attempt” indicating a malicious web server was trying to exploit a vulnerability in IE.

A review of the PCAP noted that the target client was, in fact, running Safari on MacOS X.

This indicated that even if the web server was launching a legitimate attack, the client was not vulnerable to this attempt and therefore no further action was required. This highlights the value of full PCAP, packets in related sessions that don’t trigger alerts can offer valuable insight and context to security analysts. This allows rapid determinations to be made with confidence.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

Acknowledgements

Our thanks to the Cisco team, led by Jessica Bair-Oppenheimer and Steve Fink, for the opportunity to include EndaceProbes in the Cisco Live SOC architecture. The SOC team is a collection of Cisco experts across many Cisco solutions who were a pleasure to work with and innovate with. We came away with a great appreciation for the power and ease of use of the Cisco Security tools. The close collaboration resulted in the Endace team leaving with not only ideas for further integrations and workflow improvements, but also prototype integration extensions that were developed and proved out during the SOC.

To learn more about all the ways Endace integrates with Cisco, check out: https://www.endace.com/cisco.

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/

Helping Protect Cisco Live 2025 in San Diego

Original Entry by : Michael Morris

By Michael Morris, Senior Director of Technology Alliances

Building on our collaboration in the SOC at RSAC 2025, Endace was honoured to be invited by Cisco to provide EndaceProbe packet capture and a team of threat hunters to support the Cisco Live Security Operations Center (SOC). Our primary goal was to protect the network and attendees at Cisco Live San Diego 2025, our secondary goals were to educate and innovate.

Two EndaceProbes were deployed in Cisco’s SOC-in-a-Box, continuously recording all network traffic provided over two 10G SPAN ports. We integrated with Splunk Enterprise Security and Cisco Security Cloud to deliver real-time network visibility. This enabled the SOC team to detect, locate, and address threats to attendees from external and internal sources, and helped to streamline and accelerate SOC investigation workflows.

The SOC at Cisco Live aims to protect the more than 23,000 attendees from threats on the conference network. When an attendee’s device is found to be compromised or unsecured, the SOC team makes every effort to identify, locate and help remediate the issue. The SOC team also provided public tours of the SOC to educate attendees about how the SOC is configured, the processes team members implemented, and the sorts of issues and threats that we uncovered.

Drawing on our experience at RSAC 2025 (where Endace captured 36 TB of data and 45 billion packets) we further optimized our approach for Cisco Live’s SOC.

The SOC was set up in two days, leveraging prior planning and our engineers’ experience at RSAC 2025.

At Cisco Live, Endace:

Captured 78.9 TB of traffic representing 99.5 billion packets
Reassembled more than 740,000 file objects in real time and sent 42,624 files to Splunk Attack Analyzer for further analysis
Detected 2,256 occurrences of cleartext passwords being used from 92 unique devices
Provided packet-level forensic evidence to all the SOC analysts for investigating and triaging a range of security events
Integrated with Cisco XDR, Splunk, Cisco Firepower IDS, Splunk Attack Analyser, Cisco Secure Malware Analytics, Cisco Secure Network Analytics to provide a seamless IR workflow
Innovated with new integrations that streamed additional metadata into Splunk, which sped up the investigation and remediation process

It was impressive to see the whole team of Cisco and Endace Engineers pivoting to the packet data to investigate an entire range of security incidents. Inspecting the packet data enabled fast decisions because the team could see exactly what was happening on the wire before, during, and after any event. As they say, PCAP or it didn’t happen!

Having complete packet data available at the SOC team’s fingertips really highlighted the incredible value of always-on packet capture; recording every packet and delivering complete, real-time visibility into threats, performance issues, and anomalous behaviour across the network.

There is an infectious energy that we get when working in a SOC environment like this, with Engineers from different teams and companies collaborating to track down threats, and share ideas on how to improve the SOC. The Endace team came away with many learnings that will improve EndaceProbe and our partner integrations.

We look forward to being a part of the SOC again at some more big upcoming events – stay tuned, details coming soon.

Read more about the SOC at CLUS 25 on Cisco’s Blog here:

https://blogs.cisco.com/security/cisco-live-san-diego-2025-soc

For more blogs in our Endace SOC series, see here:
https://blog.endace.com/tag/soc/