By Andreas Löf, Software Manager, Endace
Endace supported Splunk in two different ways in the Agentic Security Operations Center (SOC) at Splunk .conf26. Firstly, capturing and recording every packet transmitted across the Spunk .conf network to create a source of ground truth so all events can be verified. Secondly, creating rich metadata ingested by Splunk and used for detection and investigation in the Agentic SOC.
During Splunk.conf26 we expanded on the data the EndaceProbe Appliances delivered to Splunk by enabling full Suricata detections and expanding the pre-existing Zeek detections we had from previous events.
These expanded rulesets allowed us to cross-correlate events between Zeek, Suricata, and Cisco Secure Firewall Threat Intelligence. Each one of these sources provides a part of the picture. Correlating them gave us better quality detections and enabled more robust investigations. EndaceVision™️ and embedded Wireshark™️ then provided the ability to evaluate findings by analyzing the actual packet evidence.
Real-world Investigation Scenario
During .conf26, there was an analyst’s queue in Splunk Enterprise Security (ES) of alerts that had been generated. The AI agents would then investigate the different events, classify them, and suggest further steps for the analysts to follow up for the event.
At .conf26, the AI agent flagged and investigated multiple instances of hosts connecting to a specific external IP address that was deemed potentially risky. We verified the agent’s findings and discovered the hosts were connecting to Baidu, often known as the Google of China. As .conf26 is attended by people from all over the world, there’s nothing inherently risky with visitors connecting to Baidu. As a part of the investigation, we verified that the connections came only from hosts on the attendee network – no instances originated from conference infrastructure – and that there was no sign of C2 traffic. We also confirmed the IP address was genuinely registered to Baidu and wasn’t spoofing a Baidu host.
Further examination of the traffic showed us there were a multitude of hosts connecting to the IP address flagged by the AI agent. Analyzing all the related traffic on the EndaceProbe showed that it was genuine traffic, primarily encrypted.
The EndaceProbes were also able to identify most of the traffic in the above screenshot as Baidu traffic, through detection rules built into the EndaceProbe’s Deep Packet Inspection (DPI) engine.
We then further verified the traffic by looking at the raw packets in Wireshark, where we could see that the traffic was to a server presenting itself as baidu.com, as can be seen from the Wireshark screenshot below.
This then allowed us to confidently move the investigation back into Splunk and look at all alerts that had been generated around these systems. We verified that all the events were going to eitherwww[.]baidu.com or sp1[.]baidu.com./
We could then confidently close these events as false positives, having verified the premise that the traffic was benign in nature.
Richer Data – More Certainty
During the preparation phase of the Agentic SOC for .conf26 we enabled more metadata collectors running on the hosted DockOS VMs on the EndaceProbes. These ran both Suricata™ and Zeek™ and performed deep packet inspection of all network traffic, without any interruptions to the capture and recording.
Comparing Zeek and Suricata shows that they have a different detection and reporting rate when operating across the same traffic.
|
System |
Event Findings |
|
Suricata |
208 |
|
Zeek |
461 |
Further investigation into the specific events (TLS inspection) shows that Zeek uncovered more events and more hosts than Suricata. However, they are different types of systems and detect different things.
This highlights the usefulness of running both systems in tandem and processing the same data. This generates richer metadata for both AI Agents and Humans to investigate and react to, as well as more detections. By correlating events across multiple sources, we get stronger evidence as to whether the event is a true positive, or a false positive. We can also more easily determine whether it is benign.
Detection Tuning
Over 30 minutes, Suricata generated 5.3 million events and Zeek about 3.1 million. For both systems, this excluded the stats reporting.
Suricata
Suricata was configured to capture traffic from eight Endace Virtual Data Acquisition and Generation (DAG) cards that received a flow-coherent, balanced traffic stream from the EndaceProbe hosting the virtual machine on which the system was running.
The events were configured to be logged into eve.json, which was then ingested into Splunk via the Splunk Log Forwarder. The logs were aggressively rotated after ingestion to manage disk space.
We enabled the open Emerging Threats rule set (approximately 55k rules) as well as most of the protocol dissectors. There’s overlap with the configuration we applied to Zeek, but the example above shows a difference in the system’s effectiveness, which makes the overlap valuable.
Zeek
Like Suricata, Zeek was also configured to capture network traffic from eight Endace Virtual DAG cards with the same setup as for Suricata.
Unlike Suricata, Zeek typically writes output from each analyzer to a separate log file. The Splunk Log Forwarder ingested these files, and they were then rotated to conserve disk space. Again, like Suricata, we enabled most of the dissectors. In addition to the protocol dissections, we had the Endace custom file extraction code and sample uploader enabled. Extracted uploaded files were then analyzed and used to generate events for any suspicious files.
Conclusions
Endace DockOS provides the ability to run both Suricata and Zeek in tandem and ingest exactly the same set of data. Both systems have some overlapping functionality but are fundamentally different in their use cases and types of alerting.
Suricata is rule and alert-based, designed to fill a role similar to Snort. Zeek, on the other hand, is designed to analyze the network traffic and generate rich metadata from it.
There is some overlap in the data Zeek and Suricate can provide, but there are also large areas where each system can create an alert/event that the other does not. We can also observe how the event rates differ from the same traffic across Suricata and Zeek. By running both systems in tandem and feeding them the same traffic from our EndaceProbes we get a better overview of the network activity, and we can pivot back to the raw packets where necessary to confirm findings or do more in-depth investigation.
As discussed in The Zero-Day Blind Spot: Why Your Agentic SOC needs Retrospective Packet Replay, it’s also possible to replay existing traffic against new detection rules to validate whether any systems have been affected by a zero-day attack. That can be extremely useful to determine whether a successful attack took place (and you have been impacted and need to respond) or, conversely, prove that no successful attacks happened.
Preparing for the next Agentic SOC
During .conf26 the human analysts made a great amount of use of the expanded event logs from Suricata and Zeek to validate AI agent investigations. In future SOCs we are looking to expand on the AI agent capabilities and the base events to make greater use of event correlation, when possible, to provide a stronger evidence chain for investigations.
This will allow AI agents to draw conclusions more quickly and easily when investigating events, and human analysts to more quickly and easily validate an AI agent’s conclusions.
Making Zeek and Suricata evidence available alongside Splunk Events gives both human analysts and AI agents much greater certainty for fast investigation and effective response. While both tools provide some common data, together they offer a very rich set of evidence, with the ability to pivot to Endace’s full packet data when needed, that gives both AI agents and human analysts deep visibility into all network activity, backed by primary, definitive, packet evidence.
Acknowledgements
Our thanks go to the Splunk .conf26 SOC team, led by Jessica Oppenheimer and Paul Pelletier, for the opportunity to integrate EndaceProbes with the Splunk .conf26 Agentic SOC architecture.
The SOC team is a collection of Cisco, Splunk and Endace experts across many domains who were a pleasure to work and innovate with, and we came away with a great appreciation for the power of the Cisco and Splunk tools.
The teams were able to prove out integration innovations and test them in earnest in a real-world environment in preparation for making them generally available to the market.
More from Endace, Splunk and Cisco in the SOC series
Read all about the Agentic Security Operations Center at Splunk .conf26, including an overview from Jessica Oppenheimer, and posts and use cases from Splunk and Cisco SOC team members:
Read about the Endace team’s experience working in the SOC at Splunk .conf26:
- Michael Morris (Endace): Why Full Packet Capture Becomes Mission-Critical in the Agentic SOC
- Andreas Löf: (Endace): Richer Data for the Agentic SOC
- Elliott Hinson (Endace): Letting the Agentic SOC Handle the Drudgery So the Humans Can Dig Deeper
- Richard Marsh (Splunk), Anantha Srinivasan (Endace) and Arunachalam Ganesan (Endace): From Novice to Senior: How an MCP-Enabled Capture Appliance Changes Threat Hunting
- Adam Kilgore (Cisco), Anantha Srinivasan (Endace) and Arunachalam Ganesan (Endace): The Zero-Day Blind Spot: Why Your Agentic SOC needs Retrospective Packet Replay
For more Endace blogs in our SOC series, see here:
https://blog.endace.com/tag/soc/
Cisco Event SOC Website
Visit Cisco’s Event SOC website for full details of the SOC-in-a-Box setup, and download the detailed architecture blueprint and report by Jessica Oppenheimer:
https://www.cisco.com/site/us/en/products/security/event-soc-report.html