Malware analysis

Rootkits 101

Introduction

Rootkits are a species of malicious software with incredible damage potential. Due to their sophistication and complexity, they also present a considerable challenge to cybersecurity professionals.

On compromised systems, rootkit detection and removal is difficult at best and outright impossible at its worst. This makes the early analysis of discovered rootkit software a high priority for security experts. 

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

The standard protocol for the evaluation of any rootkit involves reverse-engineering the malicious software in question. This will enable security professionals to uncover the following main aspects of a rootkit:

• The targeted filesystems and individual files
• The common registry keys used 
• The kind of data it is stealing, if any
• The potential of secondary attacks post-infection
• The threat of remote access to target systems
• The type of encryption used for backdoor access

Reverse engineering will give access to precise information regarding these characteristics of a rootkit, often within a matter of hours. 

This article will explore the key concepts associated with rootkits and reverse engineering, including types of rootkits and common techniques used by rootkits, like hooking and exploitation of interrupts. 

The basic types of rootkits 

Rootkit malware typically falls into two broad categories — user mode and kernel mode. Depending on the level of privileged ring access in an infected system and the attack vectors used, rootkits can also be further classified into at least five different subtypes. But for the most part, they all fall into either user mode, kernel mode or any hybrid variant involving aspects of the base categories. 

Rootkits originated in the early days of UNIX-based systems. They can be broadly defined as a collection of malicious software and tools used to exploit security vulnerabilities in any UNIX operating system. 

But in modern parlance, since Windows systems dominate the cyber ecosystem, rootkits have a much narrower definition — those that target Windows systems. They are divided into those that restrict themselves to the software space (user) of the OS, and those that delve into the deeper levels with direct firmware access (kernel). 

Persistent versus memory-resident rootkits

Most of the rootkits are designed to stay installed in the target system indefinitely. These are not erased when the system is rebooted. Another class of rootkits is designed to stay only for a limited time in the target system. 

These non-persistent rootkits are called memory-resident rootkits, as they restrict themselves to the RAM and not the filesystem itself. When the system is rebooted, the rootkit is also purged. They would be useful for the attackers in situations where they want to erase all record of the intrusion into the target system. 

User-mode rootkits

Rootkits designed to infect Windows systems often target a select shortlist of executable programs and system libraries. On Windows, this means that the following are most vulnerable to rootkits:

• The Secure Socket Shell (SSH) protocol
• C library implementations — the core system DLLs 

Applications in Windows frequently make system calls involving core DLLs and the SSH protocols. By modifying specific DLLs and SSH protocols, a rootkit can hijack or "hook" a routine system call made by an application. 

So whenever any application makes an API call involving the modified DLL, the rootkit code will be executed. To avoid detection, the code will usually modify the API return results. This helps the rootkit to stay out of registry keys, which would help it stay under the radar of most security and detection software. 

Kernel-mode rootkits

While user-mode rootkits restrict themselves to the outer rings (3 to 1) in a system hierarchy, their kernel-mode counterparts infiltrate deeper into the Ring-0 level. Rootkits with kernel access can do a lot of damage, since hooking here is relatively more reliable and impactful compared to higher levels in a Windows system. 

To enter the kernel space, a rootkit has several options: 

Interrupt hooking

In Windows, there are proxy implementations that act as barriers between the user space and kernel space. Called "gates," these are checkpoints that prevent user-level code from having general access down at the kernel level. 

When a rootkit code successfully hooks these interrupts, it can bypass the gate security features and implement its malicious code at the kernel mode. In newer Windows systems, interrupts have been replaced by Model Specific Registers, or MSRs. Rootkits have evolved in response, replacing interrupt hooking with MSR hooking. 

SSDT hooking

System Service Descriptor Table, or SSDT, is located in the kernel memory of Windows OS. It is essentially a function pointer table where all addresses of system call functions are located. Any modifications here would allow a rootkit to redirect system calls to its code. 

SSDT hooking works using the same common principles as the other techniques mentioned above. As in user mode hooking, the code will usually purge itself from the return results to avoid detection. 

Key takeaways

• Rootkits are extremely dangerous malware with a high damage potential
• There are two major types of Windows rootkits — user mode and kernel mode
• Both types use ordinary system functions like hooking for malicious intent
• SSH and DLL hooking is common in user-mode rootkits
• Interrupt hooking and SSDT hooking are the preferred methods at the kernel level

Sources

1. Rootkits, European Union Agency for Cybersecurity
2. Eugene Schultz, Ph.D., “Rootkits: The Ultimate Malware Threat,” The Information Security Management Handbook, Sixth Edition, Vol 2
3. Windows Rootkit Overview, Symantec
4. Malware Vs Viruses - Difference, Emsisoft Blog