Intro

Cybersecurity has dramatically changed in the past decade. The focus has been steadily shifting from protection to detection. In this process, threat hunting (TH) has played an instrumental role.

TH involves human “hunters” and has proven its effectiveness in detecting the presence of attacker tactics, techniques, and procedures (TTP) within an environment when automated detection solutions are helpless. 

In a 2020 SANS Institute survey, 93% of organizations reported a reduction in their attack surface after starting a TH program. And for 89%, TH increased detection and reduced the number of false positives. 

And it makes perfect sense. Technology created by humans is not smarter than its creators (possibly not yet). This means that only another human can stand against a human adversary.

In such a view, security personnel cannot afford to rely on their security tools, no matter how advanced they might be. Rather than wait for threats to come in, security teams must develop threat hypotheses based on knowledge of their environment and behaviors of adversaries and act on those hypotheses by actively testing them in the environment. This is the essence of threat hunting. 

Having said that, a threat hunter should still use automated threat detection solutions, and we will talk about the TH toolset below. Also, we’ll cover prerequisites, methods, and techniques for effective TH.

What is required for cyber threat hunting? 

Because there are no formal TH frameworks developed, it is difficult for organizations to create a mature security program. But, ultimately, there are three factors that determine the success of TH: the quality of the data, the tools to analyze that data, and the expertise of the analysts.

Data

Data is the primal resource of threat hunters. The more data, the better – the more results they will find. 

Threat analysts should start by collecting different types of data from their environment. Anything from authentication logs from systems and applications to network transactions, and HTTP server, netflow, and proxy records. 

Leverage information from outside your environment – security intel from external sources. According to the 2020 SANS survey, most organizations rely on external intel sources such as Threat Intelligence Providers (TIPS) and industry threat data banks. 

Of course, making sense of vast swaths of data you collect may be daunting, and this is where tools will lend you a hand.

Tools

A threat hunter is not a data scientist and does not have to become one if they want to use terabytes of collected information. Instead, tools with data science capabilities can do the job. 

On the market, there are many Threat Hunting Platforms (THPs) equipped with machine learning, AI, and statistical tools. They can automate common tasks such as producing activity summaries or finding outliers, the “weird” entities in data. 

Standard automation tools can greatly cut the time you spend on routine attacks, such as deleting customized scripts to find a compromised endpoint, deleting found malicious files, automatically restoring compromised data from a backup, etc. 

Then there are security monitoring tools, statistical analysis tools, intelligence analytics tools, SIEM systems, UEBA tools. 

Even basic tools can still be effective in hunting down an adversary. One can perform basic outlier analysis known as “stack counting” (described below) in regular Excel.

Team

But hunting is a proactive activity. Tools may only give a clue where a potential problem may hide. A threat hunter then takes their results and conducts hypothesis-based investigations. But a more difficult job of a threat hunter is to find what automated alerting systems have missed. An organization would need a team of analysts with a very good level of expertise in order to do that. 

In the mentioned above survey, SANS observed that more and more organizations are forming in-house TH teams. Such teams can work closely with the internal or external Security Operation Center (SOC) and collect data from various on-premises and cloud sources in order to have a deep understanding of their environment. 

It may be hard to find and hire the talent with the necessary skillset, but threat hunters can take training courses from organizations such as SANS and Certified Cyber Threat Hunter (CCTH) courses from various providers. 

Coming back to the 2020 SANS survey, 21% of organizations, having a hard time attracting qualified threat hunters, decided to outsource their threat hunting activities. 

Determine your hunting maturity, prepare, and plan ahead

The efficiency and efficacy of your TH depend on your TH maturity. Your TH capabilities, techniques, and data available will determine how mature your organization is in terms of threat hunting. 

The SANS Institute has developed a TH maturity model with five levels of proactive detection capability. Their maturity model is a good tool that security managers can use for planning and developing a TH roadmap. 

It is essential to know your maturity level and continuously improve it. When asked to grade themselves using this model, only 29% of organizations considered themselves mature or very mature. Knowing your maturity level will help you better plan your threat hunting strategy.

To determine your current level of hunting maturity start by answering a list of questions prepared by the Threat Hunting Project or do a self-assessment test by AT&T Cybersecurity. 

Hypothesize

The next step is to get to know your enemy in order to know what to look for in the data you will be collecting. Learn about your adversaries’ Tactics, Techniques, and Procedures (TTP), analyze signatures, indicators of compromise (IoC), known behaviors. 

Developing hypotheses is an important next step. Hypothesizing is included in the second stage of the SANS Institute’s Threat Hunting Practical Model because it helps to uncover visibility gaps. 

To generate hypotheses, threat hunters should look at their environment from the perspective of an attacker and ask themselves questions like “If I were an attacker, what would I do? What assets would I try to access?” Or look at any threat report and ask yourself “Is this possible in my environment? Could the same vulnerability be applied to my system?”

Once you have a good number of hypotheses, you can start testing them and developing more targeted hypotheses. 

Choose the right threat hunting technique

After all the preparation and planning, you should have a broad view of your security landscape. Now, you need to narrow your focus from there, drill down into the data. 

Among all the TH techniques used by hunters to investigate a hypothesis, we will describe the five most common ones: baselining, searching, clustering, grouping, and stacking.

Baselining

It is one of the most important threat hunting techniques. Its aim is to understand what is “normal” within your organization, and then examining deviations from this. It can be a laborious task, but the trick is to know how noisy or quiet something is in your environment. Then if you see an anomaly, it can indicate an intrusion.

Searching

When searching, you will comb through data like logs, alerts, system events, full packet data, flow records, and memory dumps for things that could indicate malicious activity – artifacts or patterns. Finely defined search criteria that are not too broad and not too narrow will produce a more manageable number of results. Even so, this technique requires a significant time investment.

Clustering

This step involves isolating clusters of similar data points with similar characteristics from a larger dataset. It can accurately find such behaviors as an unusually big number of instances of an occurrence, and thus very helpful in outlier detection. 

Grouping

Involves searching for multiple unique artifacts that appear together using certain search criteria. Unlike in Clustering, you will be searching for a particular set of artifacts that are already suspicious to you. Groups of these artifacts may be an attacker’s TTP.

Stacking

Involves working with a particular type of data, for example, a list of all running processes or endpoints of a function. The analyst then counts the number of occurrences of every process and tries to find outliers. Targeted malware will be at a lower frequency of occurrence. It is most effective when dealing with filtered inputs and is less effective when dealing with large datasets. 

Conclusion

While TH is a new field, 65% of organizations said they already had some threat hunting program, while 29% would have one in the next year. 

These are good numbers. Yet, every day brings reports of new data breaches, hacks, ransomware, compromises… 

We see the need for increasing the effectiveness of TH programs and the TH maturity of organizations. Hopefully, very soon new frameworks for implementing threat hunting programs will become available. Every organization will then start benefiting from this novel, yet powerful practice of proactive, hypothesis-driven threat discovery which is threat hunting.