Digital forensics

Rootkit Detector Features: Malicious System Threads and Debug Registers

December 16, 2011 by Arvind Doraiswamy


In my last article, we’d discussed the most important ways in which a rootkit enters a system and subsequently masks its presence so it isn’t detected. We’d also looked at two popular rootkit detectors in Tuluka and Gmer, and discussed what rootkit masking techniques they are able to identify and alert the end user. In this article, we’ll discuss two more features that these rootkit detectors possess: detecting both malicious system threads and the usage of debug registers by malware. Let’s go!

Identifying malicious system threads…

When you click on a benign Executable to launch it and then look at task manager, you’ll most probably see an entry there. That means that by clicking on the icon for calc.exe, you indirectly said that you wanted to launch a process. Of course, all you wanted to do was add two hexadecimal numbers but the operating system doesn’t understand all that. All it knows is that it has to launch a process and then assign something called a PID [ProcessID] to it, in case it wants to interact with it in the future. Now let’s take an example of a webserver whose binary is called apache.exe. Once it is launched, the web server is running and multiple users will want to access it. Since Apache has to handle numerous requests in parallel (among many other things) and needs to do it all very fast, it will use threads. Each thread is a mini process which will handle a certain important part of Apache’s operations.

Now a rootkit is no different. It will launch, then start spawning threads for its various operations. It’ll have one thread to establish a connection to an IRC server, one thread to copy files on to disk, one thread to load a driver into memory…just about anything really. The rootkit could create threads which belong to their own process, or create threads in the context of any other running process, using a technique called ‘Thread Injection’.

Last time, we saw that rootkit detectors to detect hidden processes. Here, we see that both Tuluka and Gmer have features where they detect hidden threads as well. Here is a screenshot of Tuluka before I installed a rootkit on the system:

Here is a screenshot of Tuluka after I installed a rootkit.

Here is a screenshot of Gmer after I installed a rootkit.

You’ll notice that both Tuluka and Gmer detected a thread which had its starting address at 866B6D00 – confirming the fact that there is indeed a rootkit on the system.

Identifying rootkits that use debug registers

If you’ve debugged anything at any time (even a “Hello World” program), you would have surely heard of breakpoints. You set a breakpoint to stop execution at a particular point while you check the behaviour of the program and where it’s going wrong, or what it’s writing to a particular register or memory location. When you want to understand how a particular piece of malware is working under the hood, you load it in a debugger like Olly. Almost certainly you will set a breakpoint at some point in the code, unless you’re a super geek who can understand dead disassembly listings :). So this breakpoint that you set is a software breakpoint. In a nutshell, when you set a software breakpoint, the existing assembly instruction is replaced by an interrupt (INT 3), which the debugger recognizes and halts the program. Once you want to proceed with debugging the program, the original instruction is rewritten to that address and execution continues: make sense?

Now all this works perfectly if the code that you’re debugging is static and does not change. If, however, the code is self-modifying (i.e. it changes itself at runtime), a software breakpoint is not going to be of much help. Why? Think about it: you set a breakpoint at Address A – Address A is overwritten with INT 3. Now you run the malware inside the debugger. The code starts modifying itself; that’s how the malware was written to make reversing harder. As it is modifying itself, it is also going to overwrite the breakpoint that you just set! This is an obvious problem, as we won’t break where we wanted to.

To address this problem, we use Hardware Breakpoints. These breakpoints are NOT overwritten even if the code is self-modifying. You can, however, only set a maximum of four hardware breakpoints on your x86 system; that is all there is support for.

Hardware breakpoints are stored in registers called Debug Registers [DR0 – DR7]. Yes, there are eight registers, but as I mentioned earlier, only four can be used to store your breakpoints. The Debug Register DR7 stores details regarding the hardware breakpoints set. So in effect, when you set a hardware breakpoint, you fill up one of the registers DR0 – DR3 with the address that you want to break. In addition, DR7 also updates itself, setting or resetting bits depending on which registers store the hardware breakpoints.

Some malware manipulate debug registers, clearing them so that no hardware breakpoints are triggered at all. This results in you not being able to reverse the program unless you bypass this particular protection. I couldn’t find a sample of malware to show you a couple of screenshots on how Tuluka detects changes made to Debug registers. However, Tuluka does claim to have such a feature. To compensate partly 🙂 for not showing you a screenshot, I’ll show you how exactly Debug registers are filled or cleared in OllyDbg. Malware should behave in a very similar way.

Here’s a screenshot with no breakpoints set and all Debug registers (last 7 lines, upper right pane) set to 0.

Now let’s set a hardware breakpoint on an address and look at the registers again.

You will immediately see that there is a hardware breakpoint set and the contents of DR0 and DR7 have changed. If you delete the hardware breakpoint, all the values will be zeroed out again. Malware will behave in ways similar to these.


In this shorter follow-up article, we’ve discussed two other ways in which rootkits function. With that, we’ve pretty much covered all the major features that Tuluka has to offer. Sure, there may be ways in which rootkits hide themselves (and which Tuluka does not detect) but that’s outside of the scope of this little series.

In future articles, I’ll try and look at how we can detect all these rootkits using memory analysis tools or, maybe discuss HOW exactly all these rootkit detectors work under the hood. For now though, that’s it from me 🙂


  • Debug registers –
  • Breakpoints –
  • Process vs Thread –
  • Anti Debug Reference –
Posted: December 16, 2011
Arvind Doraiswamy
View Profile

Arvind is an Information Security Professional with 6 years of experience in system, network and Web application penetration testing. He is currently a security researcher at Infosec Institute. In addition, he freelances in information security and product development [Perl, Ruby on Rails], while spending a lot of time learning more about malware analysis and reverse engineering.