The Importance of Network Data to Threat Hunting (Part 3)

Original Entry by : Robert Salier

Frameworks and Regulations

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceIn this, the third article in our series on threat hunting (see here for Part 1 and Part 2), we explore the frameworks and regulations most relevant to threat hunting.

These tend to fall into two categories: those that address cybersecurity at a governance level, and those that facilitate insight into individual attacks and help formulate appropriate defense actions.

Governance Level Frameworks and Regulations

The regulatory environment influences threat hunting, and cyber defense in general. In many countries, regulations impose obligations on disclosure of breaches, including what information must be provided, when, and to which stakeholders. This influences the information that an organization needs to know about a breach, and hence its choice of strategies, policies, processes and tools. These regulations generally require companies to disclose a breach to all customers that have been affected. However if an organization cannot ascertain which customers were affected, or even if any customers were affected, then they may need to contact every customer. The only thing worse than having to disclose a breach is having to disclose a breach without being able to provide the details your customers expect you to know.

There are a also a number of frameworks addressing cybersecurity at the governance level, which in some cases overlap with regulations, dealing with many of the same issues and considerations. Collectively, these frameworks and regulations help to ensure organizations implement good strategies, policies, processes and tools, e.g. …

  • Which systems and data is most important to the organization
  • What Information security policies should be in place
  • How cybersecurity should be operationalized (e.g. what organizational structure, security architecture and systems are most appropriate for the organization)
  • Incident management processes
  • Best practice guidelines

Prevalent frameworks and regulations include…

  • ISO 27000 Series of Information Security Standards
    A comprehensive family of standards for information security management, providing a set of best practices for information security management. Maintained by the International Standards Organization, it has been broadly adopted around the globe.
  • NIST Special Publication 800-53
    A catalogue of security and privacy controls for all U.S. federal organizations except those related to national security.
  • NIST Cybersecurity Framework
    A policy framework for private sector organizations to assess and improve their ability to prevent, detect, and respond to cyber attacks. It was developed for the USA, but has been adopted in a number of countries.
Frameworks to Characterize Attacks and Facilitate Responses

A number of frameworks have been developed to help describe and characterize attacker activity, and ultimately facilitate defense strategies and tactics.

Prevalent frameworks include…

  • Cyber Kill Chain
    Developed by Lockheed Martin, this framework was developed from a “kill chain” framework developed for military attack and defense. It decomposes a cyber attack into seven generic stages, providing a framework for characterising and responding to attacks. Refer to this Dark Reading article for some discussion on the benefits and limitations of this framework.
  • Diamond Model
    This model describes attacks decomposing an attack into four key aspects, i.e. details of the adversary, their capabilities, the infrastructure they used, and the victim(s). Multiple attack diamonds can be plotted graphically in various ways including timelines and groupings, facilitating deeper insight.
  • Mitre Att&ck
    Developed by Mitre, Att&ck stands for “Adversarial Tactics, Techniques, and Common Knowledge”. It is essentially a living, growing knowledge base capturing intelligence gained from millions of attacks on enterprise networks. It consists of a framework that decomposes a cyber attack into eleven different phases, a list of techniques used in each phase by adversaries, documented real-world use of each technique, and a list of known threat actor groups. Att&ck is becoming increasingly popular, used by and contributed to by many security vendors and consultants.
  • OODA Loop
    Describes a process cycle of “Observe – Orient – Decide – Act”. Originally developed for military combat operations, it is now being applied to commercial operations.

The Importance of Network Data to Threat Hunting (Part 2)

Original Entry by : Robert Salier

Threat Hunting in Practice

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceHunting for security threats involves looking for traces of attackers in an organization’s IT environment, both past and present. It involves creativity combined with (relatively loose) methodologies and frameworks, focused on outsmarting an attacker.

Threat Hunting relies on a deep knowledge of the Tactics, Techniques and Procedures (TTP’s) that adversaries use, and a thorough knowledge of the organization’s IT environment. Well executed threat hunts provide organizations with deeper insight into their IT environment and into where attackers might hide.

This, the second article in our series of blog posts on threat hunting (read Part 1 here), looks at how leading organizations approach threat hunting, and the various data, resources, systems, and processes required to threat hunt effectively and efficiently.

Larger organizations tend to have higher public profiles, more valuable information assets, and complex and distributed environments that present a greater number of opportunities for criminals to infiltrate, hide, and perform reconnaissance without detection. When it comes to seeking out best practice, it’s not surprising that large organizations are the place to look.

Large organizations recognize that criminals are constantly looking for ways to break in undetected and that it is only a matter of time before they succeed if they haven’t already. While organizations of all sizes are being attacked, larger organizations are the leaders in this proactive approach to hunting down intruders, i.e. “threat hunting”. They have recognized that active threat hunting increases detection rates over-relying on incident detection alone – i.e. waiting for alerts from automated intrusion detection systems that may never come.

Best practice involves formulating a hypothesis about what may be occurring, then seeking to confirm it. There are three general categories of hypothesis:

  • Driven by threat intelligence from industry news, reports, and feeds.
    e.g. newsfeeds report a dramatic increase in occurences of a specific ransomware variant targeting your industry. So a threat hunt is initiated with the hypothesis that your organization is being targeted with this ransomware
  • Driven by situational awareness, i.e. focus on infrastructure, assets and data most important to the organization.
    e.g. a hypothesis that your customers’ records are the “crown jewels”, so hackers will be trying to gain access to exfiltrate this data

Having developed a hypothesis as a starting point, leading organizations rely on a range of tools and resources to threat hunt efficiently and effectively:

Historic Data from Hardware, Software and the Network
  • Infrastructure Logs from the individual components of hardware and software that form your IT environment, e.g. firewalls, IDS, switches, routers, databases, and endpoints. These logs capture notable events, alarms and other useful information, which when pieced together can provide valuable insight into historic activity in your environment. They’re like study notes that you take from a text book, i.e. highly useful, but not a full record, just a summary of what is considered notable. Also, be wary that hackers often delete or modify logs to remove evidence of their malicious activity.
  • Summarized network data (a.k.a. “packet metadata”, “network telemetry”). Traffic on network links can be captured and analysed in real time to generate a feed of summary information characterizing the network activity. The information that can be obtained goes well beyond the flow summaries that Netflow provides, e.g. by identifying and summarizing activity and anomalies up to and including layer 7 such as email header information and expired certificates. This metadata can be very useful in hunts and investigations, particularly to correlate network traffic with events and activity from infrastructure logs, and users. Also, unlike logs, packet metadata cannot be easily deleted or modified.
  • Packet level network history. By capturing and storing packets from a network link, you have a verbatim copy of the communication over that link, allowing you to see precisely what was sent and received, with zero loss of fidelity. Some equipment such as firewalls and IDS’s capture small samples of packets, but these capture just a fraction of a second of communications, and therefore must be automatically triggered by a specific alarm or event. Capturing and storing all packets (“full packet capture”, “100% packet capture”) is the only way to obtain a complete history of all communications. Historically, the barriers to full packet capture have been the cost of the required storage and the challenge of locating the packets of interest, given the sheer volume of data. However, recent advances in technology are now breaking down those barriers.
Baselines

Baselines are an understanding of what is normal and what is anomalous.
Threat hunting involves examining user, endpoint, and network activity, searching for IoA’s and IoC’s – i.e. “clues” pointing to possible intrusions and malicious activity. The challenge is knowing which activity is normal, and which is anomalous. Without knowing that, in many cases, you will not know whether certain activity is to be expected in your environment, or whether it should be investigated.

A Centralized Location for Logs and Metadata

Because there are so many disparate sources of logs, centralized collection and storage is a practical necessity for organizations with substantial IT infrastructure. Most organizations use a SIEM (Security Information and Event Manager), which may have a dedicated database for storage of logs and metadata, or may use an enterprise data lake. SIEMs can correlate data from multiple sources, support rule-based triggers, and can feature Machine Learning algorithms able to learn what activity is normal (i.e. “baselining”). Having learned what is normal, they can then identify and flag anomalous activity.

Threat Intelligence

Threat intelligence is knowledge that helps organizations protect themselves against cyber attacks. It encompasses both business level and technical level detail. At a business level this includes general trends in malicious activity, individual breaches that have occurred, and how organizations are succeeding and failing to protect themselves. At a technical level, threat intelligence provides very detailed information on how individual threats work, informing organizations how to detect, block, and remove these threats. Generally this comes in the form of articles intended for consumption by humans, but also encompasses machine-readable intelligence that can be directly ingested by automated systems, e.g. updates to threat detection rules.

Frameworks and Regulations

The regulatory environment influences threat hunting, and cyber defense in general. In many countries, regulations impose obligations on disclosure of breaches, including what information must be provided, when, and to which stakeholders. There are a also a number of frameworks addressing cyber security at the governance level, which in some cases overlap with regulations, dealing with many of the same issues and considerations. Collectively, these frameworks and regulations help to ensure organizations implement good strategies, policies, processes and tools.

In the next article in this series, we explore the frameworks and regulations that apply to threat hunting, and which ensure organizations implement appropriate strategies, policies, processes and tools.


The Importance of Network Data to Threat Hunting (Part 1)

Original Entry by : Robert Salier

Introduction to Threat Hunting

By Robert Salier, Product Manager, Endace


Robert Salier, Product Manager, EndaceCriminal hackers are stealthy. They put huge efforts into infiltrating without triggering intrusion detection systems or leaving traces in logs and metadata … and often succeed. So you need to actively go searching for them. That’s why SecOps teams are increasingly embracing threat hunting.

This is the first in a series of blog articles where we discuss various aspects of threat hunting, and how visibility into network traffic can increase the efficiency and effectiveness of threat hunting. This visibility is often the difference between detecting an intruder, or not, and collecting the conclusive evidence you need to respond to an attack, or not.

In December 2015 Ukraine suffered from a power grid cyber attack that disrupted power distribution to the nation’s citizens. Thirty substations were switched off and damaged leaving 230,000 without power.

This attack was meticulously planned and executed, with the attackers having first gained access over six months before they finally triggered the outage. There were many stages of intrusion and attack, leaving traces that were only identified in subsequent investigations. Well planned and executed threat hunting would probably have uncovered this intruder activity, and averted the serious outages that took place.

This is a good example of why, in the last few years, threat hunting has been gaining substantial momentum and focus amongst SecOps teams, with increasing efforts to better define and formalize it as a discipline. You’ll see a range of definitions with slightly different perspectives, but the following captures the essence of Threat Hunting:

The process of proactively and iteratively searching through IT infrastructure to detect and isolate advanced threats that evade existing security solutions.

There’s also some divergence in approaches to threat hunting, and in the aspects that individual organizations consider most important, but key themes are:

  • To augment automated detection, increasing the likelihood that threats will be detected.
  • To provide insight into attackers’ Tactics, Techniques and Procedures (TTP’s) and hence inform an organization where they should focus their resources and attention.
  • To identify if, and where, automated systems need updating – e.g. with new triggers.

So, threat hunting involves proactively seeking out attacks on your IT infrastructure that are not detected by automated systems such as IDS’s, firewalls, DLP and EDR solutions. It’s distinct from incident response, which is reactive. It may, however, result in an incident response being triggered.

Although threat hunting can be assisted by machine-based tools, it is fundamentally an activity performed by people, not machines, heavily leveraging human intelligence, wisdom and experience.

In the next article, we explore how leading organizations approach threat hunting, and the various data, resources, systems, and processes required to threat hunt effectively and efficiently.

In the meantime, feel free to browse the Useful References page in our Theat Hunting Section on endace.com, which contains both a glossary and useful links to various pages related to threat hunting. Below are some additional useful references.

References

(1) Threat Hunting Report (Cyber Security Insiders), p22

(2) 2018 Threat Hunting Survey Results (SANS), p13

(3) 2018 Threat Hunting Survey Results (SANS), p5

(4) Improving the Effectiveness of the Security Operations Center (Ponemon Institute), p10

(5) The Ultimate Guide To Threat Hunting, InfoSec Institute

 


Watch Endace on Cisco ThreatWise TV from RSA 2019

Original Entry by : Endace

It was a privilege to attend this year’s RSA cybersecurity event in San Francisco, and one of our top highlights was certainly the opportunity to speak to Cisco’s ThreatWise TV host Jason Wright. Watch the video on Cisco’s ThreatWise TV (or below) as Jason interviews our very own Michael Morris to learn more about how Cisco and Endace integrate to accelerate and improve cyber incident investigations.

In this short 4 minute video, Michael demonstrates how Cisco Firepower and Stealthwatch can be used together to investigate intrusion events, using Cisco dashboards and EndaceVision to drill down into events by priority and classification to show where threats come from, who has been affected and whether any lateral movement occurred, as well as conversation history and traffic profiles. Michael also explains how Cisco and Endace work together to ‘find a needle in a haystack’ across petabytes of network traffic.

A big thanks to Cisco and to Jason for giving us this spotlight opportunity. If you have any questions about how Cisco and Endace integrations can accelerate and improve cyber incident investigation, visit our Cisco partner page.


New OSm 6.5 brings ultra-fast, network-wide search to all EndaceProbe models

Original Entry by : Sebastian Mackay

OSm - Operating System for Monitoring

We are really excited to announce the release of OSm 6.5

This significant new release incorporates some major architectural changes and introduces a truly revolutionary feature – ultra-fast, network-wide search and data-mining – with the brand-new InvestigationManager™ application

Customers are always telling us how important it is to accelerate the investigation of security threats and performance issues so they can respond to them more quickly and more accurately.

InvestigationManager is a game-changer for analysts involved in the investigation process, allowing them to search across petabytes of globally-distributed Network History for specific “packets-of-interest” at lightning-speed, putting definitive evidence at their fingertips when they need it.

New Groundbreaking EndaceFabric Architecture 

Watch this short video for an overview of the architectural changes that OSm 6.5 introduces and how this new architecture underpins the amazing new, ultra-fast search capability that InvestigationManager brings to all EndaceProbe models.

InvestigationManager’s Ultra-Fast Search in Action

Watch this demo to see just how fast InvestigationManager can find specific “needle-in-the-haystack” packet from within more than a petabyte of Network History distributed across multiple EndaceProbes deployed around the world.

(Tip: prepare to be impressed!).

Want to Find Out More?

OSm 6.5 includes a number of other updates including:
• Real-time visualizations in both InvestigationManager and EndaceProbes (“Play Mode”)
• The ability to trigger, collect and export system and RAID dumps from one or more EndaceProbes at a time.

You can read more about the new features of OSm and the new InvestigationManager application on endace.com.

Or watch the video below for a deep-dive into the new features of OSm 6.5.2 and InvestigationManager and what the new ultra-fast search capability of InvestigationManager means for Threat Hunting.

How do I get hold of OSM 6.5?

OSm 6.5 is supported by all current EndaceProbe models.

The downloadable image and documentation for OSm will be available on the Endace Support Portal from early February, 2019.

If you wish to install this new release earlier, please contact your Endace account team.


Endace announced as double finalist in 2018 Computing Security Awards and UK IT Industry Awards

Original Entry by : Mark Evans

Computing Security Awards 2018

Our EndaceProbe™ Analytics Platform has been announced as a double finalist in the ‘Network Security Solution of the Year’ and the ‘Enterprise Security Solution of the Year’ categories for the 2018 Computing Security Awards.

The Computing Security Awards started in 2010 to recognize security champions and solutions throughout the UK IT industry. The winners of the awards will be announced on the 11th October at an awards ceremony dinner at the Radisson Blu Edwardian Hotel, London.

You can vote in all categories of the Computing Security Awards, here: http://www.computingsecurityawards.co.uk/?page=csa2018vote. If you can spare a minute to vote, we’d be very grateful for your support!

UK IT Industry Awards 2018

Endace has also been announced as a finalist in the 2018 UK IT Industry Awards in the ‘Security Innovation of the Year’ category.

The UK IT Industry Awards celebrates IT excellence and focuses on the contribution of individuals, projects, organizations and technologies that have excelled in the use, development and deployment of IT in the past 12 months. The award ceremony for the UK IT Industry Awards will take place on the 14th November in Battersea, London.

The EndaceProbe is the industry’s only, truly, open packet capture platform, allowing both hosting of and integration with commercial, open-source and custom analytics applications.

You can learn more about Endace’s network monitoring products, analytics platform, and network packet history recording solutions here.


Endace Team Winners at Cyber Challenge

Original Entry by : Mark Evans

This year’s Cyber Challenge, run by the University of Waikato, attracted a wide range of participants from around the country with participants attending from schools and tertiary institutions as well as industry.

The youngest competitor was just twelve-years-old!

As a longtime sponsor of the event, Endace this year decided to enter three teams in the challenge.

Teams participated in a series of challenges, including capture-the-flag challenges and a drone challenge. Endace team, Team Dark Arts (consisting of Deepak Ramaprasad, Leo Liu and Dr. Andreas Löf) took line honours after Rounds 1 and 2, taking out the prize for the winning team for Rounds 1 and 2.

Team Dark Arts donated their prize to the runners-up after Rounds 1 and 2, Team Arcton (Jeremy Symon and Grady Hooker, another industry team). Which then graciously donated the prize to the third-placed team, Team Purple (James Donaldson and Tom Crisp), another industry team.

However overall glory, and bragging rights for 2018, fell to Team Elliptic Curveballs (Jacob Cheatley, Michael Jang and Logan Krippner – all University of Waikato computer science students). Congratulations guys!

Associate Professor, Dr. Ryan Ko, Head of Cyber Security Lab and Director, New Zealand Institute for Security and Crime Science, said this year’s Cyber Challenge was the toughest yet.

“The standard of competitors keeps getting higher, but we’re also creating more challenging tasks. It speaks well for the future cybersecurity landscape, and the University of Waikato,” Dr. Ko said.

Congratulations to all the participants in this year’s Cyber Challenge, and our thanks to the team at the University of Waikato for hosting the event.

And to all the Endace participants, from all the teams: “well done, you did us proud!”


Black Hat Europe 2017: Where the Best Minds in Cybersecurity Meet

Original Entry by : Leah Jones

Christmas and New Year may be approaching fast, but the ever-changing and unpredictable world of Information Security continues at full speed.

From the 4th-7th of December, we’ll be exhibiting at Black Hat Europe at the ExCel, London.

Attended by cybersecurity professionals and enthusiasts from around the world, Black Hat Europe 2017 will bring the best and brightest in the industry together to share information on the latest research, developments and trends.

We’ll be at our at stand (booth 201) throughout the event to answer questions and to share thoughts and ideas with attendees, particularly on the major breaches of recent years and the impending GDPR legislation. With the May 2018 deadline not far away, organizations need to be aware of how to respond to potential data breaches quickly or face hefty fines if they are inadequately prepared.

Some of the major breaches that we’ll be discussing include:

  • Equifax, a victim of one of the largest hacks in recent memory. The company took two months to admit that the breach had taken place. Post-GDPR, Equifax would need to reduce their identification and reporting time from two months to just 72 hours.
  • Deloitte, where a cyberattack on the company’s Azure-hosted email server’s administration account resulted in confidential documents and emails being stolen. To prepare for GDPR, cloud providers need to prioritize network visibility, something that current cloud software structures often hinder.
  • TalkTalk, which announced in 2015 that a breach had taken place, erred on the side of caution by “over-reporting”, later discovering the breach was not as bad as first thought. Under GDPR, more companies may be inclined to over-report, given potential fines of up to 4% of their global revenue for under-reporting. In a post-GDPR world, precision in post-breach analysis and forensics is essential.

We’ll be demonstrating how our EndaceProbe Network Recorders can be integrated with security tools from partners like Cisco, Splunk, Plixer and Palo Alto Networks to accelerate the investigation of security alerts and help companies to identify and respond to intrusions before they can escalate into a major breach.

We’ll also be talking to attendees about why recording their network traffic provides the only truly reliable evidence for conclusively determining the cause and scope of security intrusions and breaches.

Attending Black Hat London 2017 and want to learn more about Endace? Visit our exhibition at booth 201 and meet our team. If you’re unable to attend Black Hat, visit our website to learn more about Endace and our EndaceProbe Network Recorders . Or follow us on Twitter or LinkedIn


How to protect against nation state attackers

Original Entry by : Mark Evans

“One of my worst nightmares [as an attacker] is that out-of-band network tap that really is capturing all the data, understanding anomalous behaviour going on. And someone’s paying attention to it.”
Rob Joyce, NSA: “Disrupting Nation State Attackers, Jan 2016” (22:10)

It’s great to see the efficacy of packet capture and network recording acknowledged by such an eminent cybersecurity Tsar as Rob Joyce.

If you haven’t already seen his video presentation on Disrupting Nation State Attackers, it’s well worth a watch. Before being shoulder-tapped to take up his new role as a cybersecurity advisor to Trump’s National Security Council, Joyce headed up the Tailored Access Operations division of the NSA.

The NSA’s TAO division is responsible for “providing tools and expertise in computer network exploitation to deliver foreign intelligence.” In other words, it is responsible for finding, and taking advantage of, the very network vulnerabilities that we’re all trying to protect against.

In his presentation at the Usenix Enigma conference last year, Joyce outlined key steps organizations can take to protect themselves against the sort of sophisticated techniques employed by Nation State attackers and criminal elements looking to attack your network.

Much of his advice is practical common sense. Know everything on your network, understand it, and update and patch everything. We all know this is critical, but all too often it doesn’t happen. Take patching for example. Joyce says that, in his experience, many organizations undertake security audits to identify known vulnerabilities, but frequently have still not fixed those vulnerabilities by the time the next audit rolls around months later.

Joyce also explodes a common myth – that sophisticated intruders rely on zero day threats. In fact, he says, zero day threats are far from being biggest danger to corporate networks. For any large network, he says:

Persistence and focus will get you in and achieve that exploitation without the zero days. There’s so many vectors that are easier, less risky and quite often more productive.

The cause of most intrusions, says Joyce, come down to one of things (the “Big Three”):

  • Email:  “a user clicked on something they shouldn’t have”
  • Malicious websites“they’ve gotten to a malicious website … and it’s either executed or they’ve run content from that website.”
  • Removable media – “where a user inserted contaminated media“. [As an aside, someone once told me the easiest way to get malware into an organization is to load it on a USB stick labelled “Payroll”, drop it in the carpark and leave the rest to curiosity!].

Joyce outlines the importance of making sure that sources of information about activity on the network – such as log files or network packet captures – are actually being monitored. “You’d be amazed at incident response teams that go in and there’s been some tremendous breach .. Yep, there it is right there in the logs.”

But perhaps the best piece of strategic advice he offers is this:

“Consider that you’re already penetrated. Do you have the means and methods to understand if somebody’s inside your network?”

That change in focus is important. Statistics show intrusions are becoming increasingly commonplace. Once organizations move from “we need to make sure we’re not penetrated” to “maybe we already are penetrated” they start to understand what tools, skills and processes they need to put in place to identify intrusions and stop an initial penetration from going on to become a more serious data breach. Or, if they have already been breached, what do they need to make sure they can identify how it happened and what was compromised?

Joyce’s presentation is a salient reminder that ensuring the basics of network security hygiene is critical. And that the battle to defend against attackers is an ongoing one. As fast as you tighten up your security, new vulnerabilities emerge that put you at risk.

Take a look at the video. You’ll find it’s 30 minutes of your time very well spent!

Cybersecurity Resources

Some of the useful resources that Joyce discusses and recommends are listed below

NOTE: The two links to the IAD site above require installing the DoD Root CA Certificates to avoid getting an “untrusted website” notification. More information here.