The Importance of Network Data to Threat Hunting (Part 3)

Original Entry by : Robert Salier

Frameworks and Regulations

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceIn this, the third article in our series on threat hunting (see here for Part 1 and Part 2), we explore the frameworks and regulations most relevant to threat hunting.

These tend to fall into two categories: those that address cybersecurity at a governance level, and those that facilitate insight into individual attacks and help formulate appropriate defense actions.

Governance Level Frameworks and Regulations

The regulatory environment influences threat hunting, and cyber defense in general. In many countries, regulations impose obligations on disclosure of breaches, including what information must be provided, when, and to which stakeholders. This influences the information that an organization needs to know about a breach, and hence its choice of strategies, policies, processes and tools. These regulations generally require companies to disclose a breach to all customers that have been affected. However if an organization cannot ascertain which customers were affected, or even if any customers were affected, then they may need to contact every customer. The only thing worse than having to disclose a breach is having to disclose a breach without being able to provide the details your customers expect you to know.

There are a also a number of frameworks addressing cybersecurity at the governance level, which in some cases overlap with regulations, dealing with many of the same issues and considerations. Collectively, these frameworks and regulations help to ensure organizations implement good strategies, policies, processes and tools, e.g. …

  • Which systems and data is most important to the organization
  • What Information security policies should be in place
  • How cybersecurity should be operationalized (e.g. what organizational structure, security architecture and systems are most appropriate for the organization)
  • Incident management processes
  • Best practice guidelines

Prevalent frameworks and regulations include…

  • ISO 27000 Series of Information Security Standards
    A comprehensive family of standards for information security management, providing a set of best practices for information security management. Maintained by the International Standards Organization, it has been broadly adopted around the globe.
  • NIST Special Publication 800-53
    A catalogue of security and privacy controls for all U.S. federal organizations except those related to national security.
  • NIST Cybersecurity Framework
    A policy framework for private sector organizations to assess and improve their ability to prevent, detect, and respond to cyber attacks. It was developed for the USA, but has been adopted in a number of countries.
Frameworks to Characterize Attacks and Facilitate Responses

A number of frameworks have been developed to help describe and characterize attacker activity, and ultimately facilitate defense strategies and tactics.

Prevalent frameworks include…

  • Cyber Kill Chain
    Developed by Lockheed Martin, this framework was developed from a “kill chain” framework developed for military attack and defense. It decomposes a cyber attack into seven generic stages, providing a framework for characterising and responding to attacks. Refer to this Dark Reading article for some discussion on the benefits and limitations of this framework.
  • Diamond Model
    This model describes attacks decomposing an attack into four key aspects, i.e. details of the adversary, their capabilities, the infrastructure they used, and the victim(s). Multiple attack diamonds can be plotted graphically in various ways including timelines and groupings, facilitating deeper insight.
  • Mitre Att&ck
    Developed by Mitre, Att&ck stands for “Adversarial Tactics, Techniques, and Common Knowledge”. It is essentially a living, growing knowledge base capturing intelligence gained from millions of attacks on enterprise networks. It consists of a framework that decomposes a cyber attack into eleven different phases, a list of techniques used in each phase by adversaries, documented real-world use of each technique, and a list of known threat actor groups. Att&ck is becoming increasingly popular, used by and contributed to by many security vendors and consultants.
  • OODA Loop
    Describes a process cycle of “Observe – Orient – Decide – Act”. Originally developed for military combat operations, it is now being applied to commercial operations.

The Importance of Network Data to Threat Hunting (Part 2)

Original Entry by : Robert Salier

Threat Hunting in Practice

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceHunting for security threats involves looking for traces of attackers in an organization’s IT environment, both past and present. It involves creativity combined with (relatively loose) methodologies and frameworks, focused on outsmarting an attacker.

Threat Hunting relies on a deep knowledge of the Tactics, Techniques and Procedures (TTP’s) that adversaries use, and a thorough knowledge of the organization’s IT environment. Well executed threat hunts provide organizations with deeper insight into their IT environment and into where attackers might hide.

This, the second article in our series of blog posts on threat hunting (read Part 1 here), looks at how leading organizations approach threat hunting, and the various data, resources, systems, and processes required to threat hunt effectively and efficiently.

Larger organizations tend to have higher public profiles, more valuable information assets, and complex and distributed environments that present a greater number of opportunities for criminals to infiltrate, hide, and perform reconnaissance without detection. When it comes to seeking out best practice, it’s not surprising that large organizations are the place to look.

Large organizations recognize that criminals are constantly looking for ways to break in undetected and that it is only a matter of time before they succeed if they haven’t already. While organizations of all sizes are being attacked, larger organizations are the leaders in this proactive approach to hunting down intruders, i.e. “threat hunting”. They have recognized that active threat hunting increases detection rates over-relying on incident detection alone – i.e. waiting for alerts from automated intrusion detection systems that may never come.

Best practice involves formulating a hypothesis about what may be occurring, then seeking to confirm it. There are three general categories of hypothesis:

  • Driven by threat intelligence from industry news, reports, and feeds.
    e.g. newsfeeds report a dramatic increase in occurences of a specific ransomware variant targeting your industry. So a threat hunt is initiated with the hypothesis that your organization is being targeted with this ransomware
  • Driven by situational awareness, i.e. focus on infrastructure, assets and data most important to the organization.
    e.g. a hypothesis that your customers’ records are the “crown jewels”, so hackers will be trying to gain access to exfiltrate this data

Having developed a hypothesis as a starting point, leading organizations rely on a range of tools and resources to threat hunt efficiently and effectively:

Historic Data from Hardware, Software and the Network
  • Infrastructure Logs from the individual components of hardware and software that form your IT environment, e.g. firewalls, IDS, switches, routers, databases, and endpoints. These logs capture notable events, alarms and other useful information, which when pieced together can provide valuable insight into historic activity in your environment. They’re like study notes that you take from a text book, i.e. highly useful, but not a full record, just a summary of what is considered notable. Also, be wary that hackers often delete or modify logs to remove evidence of their malicious activity.
  • Summarized network data (a.k.a. “packet metadata”, “network telemetry”). Traffic on network links can be captured and analysed in real time to generate a feed of summary information characterizing the network activity. The information that can be obtained goes well beyond the flow summaries that Netflow provides, e.g. by identifying and summarizing activity and anomalies up to and including layer 7 such as email header information and expired certificates. This metadata can be very useful in hunts and investigations, particularly to correlate network traffic with events and activity from infrastructure logs, and users. Also, unlike logs, packet metadata cannot be easily deleted or modified.
  • Packet level network history. By capturing and storing packets from a network link, you have a verbatim copy of the communication over that link, allowing you to see precisely what was sent and received, with zero loss of fidelity. Some equipment such as firewalls and IDS’s capture small samples of packets, but these capture just a fraction of a second of communications, and therefore must be automatically triggered by a specific alarm or event. Capturing and storing all packets (“full packet capture”, “100% packet capture”) is the only way to obtain a complete history of all communications. Historically, the barriers to full packet capture have been the cost of the required storage and the challenge of locating the packets of interest, given the sheer volume of data. However, recent advances in technology are now breaking down those barriers.
Baselines

Baselines are an understanding of what is normal and what is anomalous.
Threat hunting involves examining user, endpoint, and network activity, searching for IoA’s and IoC’s – i.e. “clues” pointing to possible intrusions and malicious activity. The challenge is knowing which activity is normal, and which is anomalous. Without knowing that, in many cases, you will not know whether certain activity is to be expected in your environment, or whether it should be investigated.

A Centralized Location for Logs and Metadata

Because there are so many disparate sources of logs, centralized collection and storage is a practical necessity for organizations with substantial IT infrastructure. Most organizations use a SIEM (Security Information and Event Manager), which may have a dedicated database for storage of logs and metadata, or may use an enterprise data lake. SIEMs can correlate data from multiple sources, support rule-based triggers, and can feature Machine Learning algorithms able to learn what activity is normal (i.e. “baselining”). Having learned what is normal, they can then identify and flag anomalous activity.

Threat Intelligence

Threat intelligence is knowledge that helps organizations protect themselves against cyber attacks. It encompasses both business level and technical level detail. At a business level this includes general trends in malicious activity, individual breaches that have occurred, and how organizations are succeeding and failing to protect themselves. At a technical level, threat intelligence provides very detailed information on how individual threats work, informing organizations how to detect, block, and remove these threats. Generally this comes in the form of articles intended for consumption by humans, but also encompasses machine-readable intelligence that can be directly ingested by automated systems, e.g. updates to threat detection rules.

Frameworks and Regulations

The regulatory environment influences threat hunting, and cyber defense in general. In many countries, regulations impose obligations on disclosure of breaches, including what information must be provided, when, and to which stakeholders. There are a also a number of frameworks addressing cyber security at the governance level, which in some cases overlap with regulations, dealing with many of the same issues and considerations. Collectively, these frameworks and regulations help to ensure organizations implement good strategies, policies, processes and tools.

In the next article in this series, we explore the frameworks and regulations that apply to threat hunting, and which ensure organizations implement appropriate strategies, policies, processes and tools.


The Importance of Network Data to Threat Hunting (Part 1)

Original Entry by : Robert Salier

Introduction to Threat Hunting

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceCriminal hackers are stealthy. They put huge efforts into infiltrating without triggering intrusion detection systems or leaving traces in logs and metadata … and often succeed. So you need to actively go searching for them. That’s why SecOps teams are increasingly embracing threat hunting.

This is the first in a series of blog articles where we discuss various aspects of threat hunting, and how visibility into network traffic can increase the efficiency and effectiveness of threat hunting. This visibility is often the difference between detecting an intruder, or not, and collecting the conclusive evidence you need to respond to an attack, or not.

In December 2015 Ukraine suffered from a power grid cyber attack that disrupted power distribution to the nation’s citizens. Thirty substations were switched off and damaged leaving 230,000 without power.

This attack was meticulously planned and executed, with the attackers having first gained access over six months before they finally triggered the outage. There were many stages of intrusion and attack, leaving traces that were only identified in subsequent investigations. Well planned and executed threat hunting would probably have uncovered this intruder activity, and averted the serious outages that took place.

This is a good example of why, in the last few years, threat hunting has been gaining substantial momentum and focus amongst SecOps teams, with increasing efforts to better define and formalize it as a discipline. You’ll see a range of definitions with slightly different perspectives, but the following captures the essence of Threat Hunting:

The process of proactively and iteratively searching through IT infrastructure to detect and isolate advanced threats that evade existing security solutions.

There’s also some divergence in approaches to threat hunting, and in the aspects that individual organizations consider most important, but key themes are:

  • To augment automated detection, increasing the likelihood that threats will be detected.
  • To provide insight into attackers’ Tactics, Techniques and Procedures (TTP’s) and hence inform an organization where they should focus their resources and attention.
  • To identify if, and where, automated systems need updating – e.g. with new triggers.

So, threat hunting involves proactively seeking out attacks on your IT infrastructure that are not detected by automated systems such as IDS’s, firewalls, DLP and EDR solutions. It’s distinct from incident response, which is reactive. It may, however, result in an incident response being triggered.

Although threat hunting can be assisted by machine-based tools, it is fundamentally an activity performed by people, not machines, heavily leveraging human intelligence, wisdom and experience.

In the next article, we explore how leading organizations approach threat hunting, and the various data, resources, systems, and processes required to threat hunt effectively and efficiently.

In the meantime, feel free to browse the Useful References page in our Theat Hunting Section on endace.com, which contains both a glossary and useful links to various pages related to threat hunting. Below are some additional useful references.

References

(1) Threat Hunting Report (Cyber Security Insiders), p22

(2) 2018 Threat Hunting Survey Results (SANS), p13

(3) 2018 Threat Hunting Survey Results (SANS), p5

(4) Improving the Effectiveness of the Security Operations Center (Ponemon Institute), p10

(5) The Ultimate Guide To Threat Hunting, InfoSec Institute

 


The Importance of Network Data to Threat Hunting (Part 4)

Original Entry by : Robert Salier

How Endace Accelerates Threat Hunting

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, Endace

Despite having a variety of tools at their disposal, many organizations still struggle with detecting and investigating security threats effectively and efficiently.  Inevitably, some threats are not detected because skilled hackers expend a great deal of effort avoiding security monitoring systems and removing the evidence of their activity by deleting or modifying logs and files. Even when threats are detected, organizations often lack sufficient visibility to ascertain the exact scope and nature of the threat: to be certain they have completely removed it and to be totally confident they can detect and prevent a recurrence.

This is the final post in our series on threat hunting (see here for part 1, part 2 and part 3).

In this post I take a look at how the EndaceProbe Analytics Platform can accelerate threat hunting: delivering deeper insight into network activity through rich network data that provides an independent and unadulterated view of activity in your environment.  It also explains how the EndaceProbe’s open platform approach delivers significant productivity and cost benefits, breaking down traditional barriers to affordability and practicality.

Full Packet-Level Capture of Network History

Skilled hackers (and clever malware) routinely delete or modify logs and files containing traces of their malicious activity.  However, it’s virtually impossible for them to remove traces of their presence from the traffic that traverses the network.  So, monitoring, capturing and analyzing network traffic is often the difference between being able to detect an intruder, and not, and collecting the conclusive evidence you need to address the threat, or not.

When malicious activity is detected, the next challenge is to obtain a clear picture of what has occurred.  This is critical for several reasons. Firstly, enterprises have regulatory or policy obligations such as complying with information security standards and breach disclosure regulations. Secondly, it’s critical to be able to keep stakeholders – including executive management, PR, Legal, HR suppliers, partners, and customers – informed and be able to accurately answer any questions.  And last, but not least, having a clear, unambiguous picture of what has occurred is also essential to be able to confirm that the threat has been neutralized and for you to be confident that sufficient measures are put in place to prevent a re-occurrence.

As discussed in Part 2 of this series, log files and other data sources such as flow-based network data can provide valuable insight into activity. And they might enable you to detect a threat. The problem is these data sources often don’t contain sufficient detail to enable a clear picture of exactly what happened, how it happened and what the impact is. Server and firewall logs, for example, might reveal communication between a host on your network and a malicious external host. But they can’t tell you what the actual contents of that communication were.

Capturing and storing packet history, on the other hand, gives you a verbatim copy of communications over the network, allowing you to see precisely what was sent and received with zero loss of fidelity. Packets contain all the contents: allowing accurate reconstruction of the entire conversation including file and document contents, web page interactions, emails, audio and video streams, etc.

Research report from EMA identifies packet capture as a key enabler for stronger security

Enterprise Management Associates (EMA) surveys enterprises annually to report on the strategies leading organizations are adopting to strengthen their cyber defenses. In the 2019 edition of “Unlocking High Fidelity Security”, packet capture was highlighted as a key enabler of stronger cybersecurity.

 

Download a Free Copy

 

Open Platform Approach

EndaceProbes can host a range of third party security solutions including Intrusion Detection Systems, virtual next-gen firewalls, AI-based security tools, and many other commercial, open-source or custom security and network or application performance monitoring solutions.  Because each EndaceProbe can host multiple tools, you only need to purchase and deploy packet capture hardware once.  You then have the freedom to choose best-of-breed tools, and the agility to quickly deploy new and/or updated tools without changing the underlying hardware platform.

Threat hunters can also dramatically accelerate and streamline investigations thanks to pre-built integrations between EndaceProbes and many third-party tools.  These integrations enable analysts to click on an alarm/event in any of these tools to quickly retrieve and analyze the related full packet data that is recorded on the EndaceProbes on the network.

For more details check out The Benefits of an Open Analytics Platform.

Breakthrough density and affordability

We’re very proud of our breakthrough density and price per petabyte, putting a month or more of network history within reach of many more organizations.  Our EP-9200 EndaceProbes provide 40Gbps packet capture and built-in investigation tools, hosting capacity for up to 12 applications, and a petabyte of network history storage, all in a single appliance just four rack units high.

How do we do it?  Well, it’s not just an efficient organization and economies of scale.  We have smart engineers implementing proprietary hardware, real-time storage compression, and features such as our patented Smart Truncation™.  For more, check out https://www.endace.com/endaceprobe.

Breakthrough practicality

We realize that storing network history is of limited use if it is too difficult, expensive or time consuming to extract value from it.  We knew we had to provide a way to…

  • Centrally manage estates of EndaceProbes that may be global in scale to reduce the operational cost and minimize management overheads.
  • Enable SecOps, NetOps and IT teams to quickly and easily find packets of interest from within terabytes or petabytes of data that may be distributed across a global network. And do this from a central point without having to figure out where those packets were recorded or which EndaceProbe they are stored on.
  • Meet the needs of large, complex, globally distributed networks, with the ability to scale to provide virtually unlimited storage capacity and monitor links of any speed.

So we developed the EndaceFabric™ architecture.

EndaceFabric allows multiple EndaceProbes to be deployed at various points throughout a network and seamlessly connected to form a network-wide packet capture, recording and hosting fabric.  Analysts can perform investigations and search and mine recorded Network History across multiple EndaceProbes simultaneously from a single UI.  Similarly, administrators can centrally manage estates of hundreds of connected EndaceProbes making it easy to configure, update and monitor the health and performance of the entire estate.

EndaceFabric provides more than a single pane of glass for administration, search and data-mining however.  The architecture also allows EndaceProbes to be stacked or grouped to create logical EndaceProbes capable of capturing traffic at practically any line rate with no limits to storage capacity scalability.

EndaceFabric is also the key to amazingly fast searches for packets of interest.  Due to the inherently distributed, parallel architecture, and our advanced search algorithms, search times remain constant regardless of the number of EndaceProbes involved.  A needle-in-a-haystack search for specific packets-of-interest across a hundred EndaceProbes and a hundred petabytes of network history can take just seconds.

For more details, check out https://www.endace.com/EndaceFabric, our videos describing the EndaceFabric architecture, and a demo showing our amazingly fast search.

And finally

This was the final article in our series on threat hunting, and how the Endace Analytics Platform can increase the efficiency and conclusiveness of threat hunts. We hope you found it useful?

If you’d like to find out more, please don’t hesitate to reach out to your local Endace representative, or contact us at https://www.endace.com/contact.