Threat Hunting Techniques

July 20, 2018 by

Graeme Messina

Introduction

Cyberthreat-hunting is a way of scouring through a network and finding sophisticated threats that could cause harm to systems and data, and which are not detectable to already existing security measures such as antiviruses and malware protection. Threat hunting requires advanced skills in cybersecurity, systems administration, programming and penetration testing.

Because advanced threats are usually able to sidestep traditional safeguards such as IDS systems and firewalls, threat hunters need to be able to replicate symptoms that they find on the network by performing similar attacks themselves and collect evidence as they scan through system logs and event files. The gap in security has been noticed, however, and there are many different vendors that create and supply software packages to help with the detection, removal and analysis of new threats.

Become a certified threat hunter

Learn how to find, assess and remove threats from your organization — and become a Certified Cyber Threat Hunting Professional, guaranteed!

Get Pricing

We will take a look at some of the threat-hunting techniques that are currently employed by cybersecurity professionals and how they are used to determine specific occurrences on a network.

Threat-Hunting Background

Threat hunting means different things to different people, but it is generally defined as the search for potential threats that are not yet known.

There are no “one size fits all”-type solutions in threat hunting, mainly due to the individual circumstances of each scenario where threat hunting is employed, but also because the individuals that undertake such investigations will have their own personal preferences and techniques. It is especially important to understand the manual nature of threat hunting, and that it has only recently become an option to use advanced automated processes to try and help to detect them.

But having said that, there are some key methodologies that are used, depending on the result that is required. We will take a look at some of these different methods and give a brief outline of where one might use each technique, given a specific scenario.  

How do organizations develop threat-hunting capabilities? It’s all about finding out what each malicious threat is trying to accomplish. Some threats may be reconnaissance and surveillance threats, gathering information and data so that an attack can be planned, while others can deliver malicious payloads. The key to defeating such threats is to develop and use threat hunting as a means of proactive defense and protection.

Technique: Searching

This is the first avenue that most threat hunters will start at, and by the name it is quite obvious why: you need to find a threat, and searching is the best way to do that.

The process entails the use of specialized queries that return results and artifacts. Because of the speculative nature of threat hunting, it is not always possible to know exactly what you are looking for when you start searching for threats. For this reason, it is important to not make your searches too broad, encompassing too many factors and receiving too many results. By contrast, making your search too narrow is also not ideal; you will miss out on potential threats that you did not include in the search.

Security professionals that engage in threat hunting activities need to analyze a lot of data, so knowing where to search is essential. But before any searching can start, evidence of some kind of threat must be apparent to some degree. Without this, it is impossible for an analyst to make any reasonable determinations about where to begin their search. Environmental data must be generated in order for any meaningful searching to be accomplished. Examples of data sources include:

• Flow records
• Logs
• Alerts
• System events
• Digital images
• Memory dumps

Searching through such large volumes of data is not practical without the use of specialized tools, and recent developments in heuristics, artificial intelligence and threat modeling all play a large role in the power of a threat hunter’s search. However, these tools cannot tell an investigator exactly what they are dealing with, so correlations must be found that will enable the investigator to take their search to the next level as they proceed through the evidence they’ve found.

Technique: Clustering/Cluster Analysis

Cluster analysis is a form of unsupervised machine learning (ML) that uses advanced AI search techniques to make correlations within advanced and vast arrays of data, such as log files and records from threat-hunting investigations. The cluster analysis takes the role of an ordinary analyst and compiles a report based on the parameters that have been set out. This can often involve terabytes of data, which would be either too time-consuming or too complicated for a human operator to assemble.

Clustering allows for a multi-dimensional picture to emerge from data and has the ability to find patterns, similarities and seemingly-unrelated correlations and then seamlessly stitch them together to form the basis of a threat-hunting starting point. It is from this point that an investigator will start to get a clearer picture of what is happening within the network environment, and how to proceed next.

The below picture illustrates how clustering can use multiple algorithms in the same set of data to provide different results:

Image source: http://commons.apache.org/proper/commons-math/userguide/ml.html

The key things to remember about this method of searching are: that it allows for data that do not necessarily share related points with one another to be analyzed as a single data set; and that it is especially useful with data obtained from data outlier systems, as they track huge volumes of behavioral data.

Technique: Group

Grouping is another method commonly employed by threat hunters, and it usually involves the use of multiple unique artifacts: running them through a series of elimination filters and seeing which ones appear together. This is useful, because seemingly-unrelated artifacts have the potential to be grouped together, giving the analyst a clue as to their relation to one another and to the possibility that there is interoperability between them. This would suggest that some sort of threat could be at work, and provides good clues for investigators to work with.

At this point, you may be wondering what the difference between clustering and grouping is, as the definitions seem quite similar at first glance. Grouping can be thought of as a separate step after clustering, as it uses an implicit set of search criteria against a data set that is already suspicious and of interest to the investigator. Clustering uses massive amounts of data to identify data sets that need to be further investigated with the grouping technique.

Technique: Stack Counting

Stacking is when a data set of similar or equal values is inspected by an investigator with the hope of finding similarities within the information provided. Sometimes a noticeable variance completely out of range for a particular metric is detected, which gives investigators a clue as to what could be producing such results.

The tools required to start stack counting are basic: even using a spreadsheet program like Excel can help you get started. Being able to sort and manipulate data in an organized manner is key to finding any similarities and anomalies in a large dataset.

One example might be a search for outgoing traffic on a specific port, executed from a particular host name. If you were to filter this information, you would end up with a stack of data specific to that query, helping you to quickly analyze the required data that you were after.

Conclusion

Threat hunting is a fascinating and challenging aspect of cybersecurity and requires attention to detail, creativity and vast knowledge of information technology and information security principles, systems and best practice. The avenues of attack that are being used by new generations of cybercriminals are becoming ever more difficult to detect, and many commercial solutions cannot guarantee the security of network environments alone.

Become a certified threat hunter

Learn how to find, assess and remove threats from your organization — and become a Certified Cyber Threat Hunting Professional, guaranteed!

Get Pricing

Sources

Using Cluster Analysis for Comprehensive Threat Detection, SecBi