Digital forensics

Rootkit Detection with Tuluka Kernel Inspector

Introduction

A rootkit is a piece of software that is written by someone, who at the very least, wants to spy on specific system calls made by an application, for some purpose. It'll usually hide itself from normal directory and process listings made either by native OS tools, or third party software which functions in the same way. It does this so that the user does not detect its presence and take steps to try and remove it.

Learn Digital Forensics

Build your skills with hands-on forensics training for computers, mobile devices, networks and more.

Start Learning

Learn Digital Forensics

Build your skills with hands-on forensics training for computers, mobile devices, networks and more.

Start Learning

A rootkit could infect a user's system in any number of ways from clicking on an Email attachment to visiting a malicious web page to running executables from untrusted sources. This being the case, security vendors and helpful open source developers have created software which will try their hardest to detect the presence of a hidden rootkit and hence help users keep their systems clean. In this article, what we'll do is look at how a rootkit functions. As we go along we'll also briefly check whether 2 freely available rootkit detectors – Tuluka and Gmer have the features necessary to combat rootkits.

Note though that the scope of the article is limited to rootkits in a Windows environment only.

Basics - You can't see me :)

If you wanted to do something bad and continue to do so for a long time you'd want to remain hidden. So you'd try and find out all the ways in which you can be detected, and develop techniques to counter those detection mechanisms. A rootkit follows precisely the same philosophy.

Lets take a very small example to understand the basic principle of a rootkit. Say the startingpoint of a puzzle is A and the ending point is D, with a guard at X..midway along the path. So it looks like this: A – B – C – X – D. So you'd have to pass X.. always, who will detect anything funny. Hence a way needs to be found around X, but still getting to D in the process. If you map that analogy to a rootkit's behavior, it would translate to – A rootkit must hide itself from the OS, do its dirty work, but allow (in most cases) the OS to function totally normally. It does so using a mechanism called "hooking" where in it creates a new path for itself to the final destination, bypassing any protections there are, on its way there. Lets take an example..

A little theory before we move on..

The Windows OS architecture is divided into 2 major parts(in the context of this article) – User Mode and Kernel Mode. The Windows API consists of numerous functions in user mode as well as kernel mode. A function call made by an application, is always a user mode function call. That is because it cannot directly call a kernel function; that's how the Windows OS is built. The UserMode function call executes; an internal switch is made to Kernel Mode, the Kernel Mode API is then called and executed.

The user wants to do a directory listing as usual, which results in the FindFirstFile being called in user mode. The FindFirstFile will call another function in user mode, following which a switch needs to be made to kernel mode. This switch is needed as the core functionality that lists files is NOT present in user mode, but in kernel mode. The user mode API knows the number of this Kernel Mode API but can't call it directly. So the processor picks up this number and passes just that to kernel mode. Now, with the help of this number something needs to happen in KernelMode which will find the eventual location of the KernelMode API which needs to be called.

I've taken a very high level view to this deliberately to keep it simple. We'll get into the details as we move along.

Hiding from directory, registry and process listings...User mode

A user will use Windows Explorer or CMD to view a list of files in a directory. So when you click on c:data in Explorer a call to FindFirstFile and subsequently to FindNextFile will be made, transparent to the user; thus listing all the files.

Lets now assume that a rootkit has also dumped all of its files in c:data and wants to hide it from the directory listing. The rootkit can hide itself in user mode.... or in kernel mode.

In user mode the rootkit hides itself from 1 specific executable; say explorer.exe or cmd.exe or it could do it for any software that uses FindFirstFile and FindNextFile to list files. In the first case, the rootkit could simply replace the valid cmd.exe with a "hooked copy" of cmd.exe. The hooked copy would never list files belonging to the rootkit itself.

Each time an EXE is loaded, it uses something called an IAT [Import Address Table] to find out the addresses of the various functions it needs to call for its normal execution. A rootkit will replace the address of the FindFirstFile function in the memory [IAT] of cmd.exe, with its own hooked function. Any third party tool though will be able to see the rootkit, as it is only cmd.exe which has been "hooked".

To hide itself from every possible tool, rootkits usually install their hooks in the kernel itself. We'll look at how it does that in a later section though. The rootkit will also want to hide any registry keys it creates as well as not be visible in any process listings made by native or third party Windows tools. To do this, it will need to hook the functions that are used to view registry keys or list processes.

Tuluka doesn't have a button or a direct option to only scan for hooked "File or Registry" API functions. GMER does though, you can just tick the Files and Registry options on the right hand side and scan a drive or a specific folder. Any hidden files will be uncovered. Both Tuluka and Gmer however try and find out if there were any hidden processes on the system.

Here's an example of an executable, which when run attempts to hide its files and registry keys. If you look at the screenshots below; you'll see that GMER has detected the hidden files as well as the hidden registry keys [Red Font in the screenshot]. If you try to either list the files in any file browser, without using Gmer you won't be able to.

Here is an example of a rootkit creating a process and hiding itself completely. Tuluka however detects the hidden process; its highlighted in RED.

IDT Hooks

On machines running older operating systems (Windows 2000 or earlier), whenever a switch needed to happen to kernel mode, an interrupt needed to be triggered. The processor would fire an instruction INT 0x2E which would force an interrupt to occur. This would mean that some application wanted to call a System service (run a Kernel mode API). Now there is something called the IDT or InterruptDescriptorTable which maintains a mapping of InterruptCodes --> AddressofHandlers. So 0x2E will map to the address of the code which handles SystemServices. So this means – anytime a kernel API needs to be called, which is very often, the IDT is looked up.

Now think, if a rootkit looked up the IDT, saved the existing address mapped to 0x2E and rewrote the address with its own function address...what would happen? Well, for starters, a function call would happen as normal in UserMode, like FindFirstFile. Eventually when a switch to kernel mode happens and the IDT is looked up; its the rootkit's code that will get called. The rootkit's code will eventually call the original code and manipulate its returned results in any way it chooses, before the results are eventually displayed back to the user.

In newer OS's though, there is no IDT lookup. So, from a rootkit's perspective, hooking the IDT is of limited use. However it is a technique none the less, and both Tuluka and Gmer detect IDT hooks. In Tuluka it is under the IDT tab, while in GMER you need to go to the Rootkit/Malware Tab and tick System and scan. If you want to take a look at all the existing entries in the IDT, make sure that you tick the 'Show All' box. There will be a huge number of results that will be displayed; look for the ones where the values under the Type column start with INT. Those are the IDT entries. To confirm if you're looking at the right thing, just look at the address under the value column and compare it with what you found in Tuluka. If both are the same, then you've understood things correctly.

Lets look at another very interesting example here. We've been talking about Tuluka as a rootkit detector; as it turns out Tuluka inserts an IDT hook itself :). So lets take a look at a screenshot of GMER scanning the system before Tuluka was started and note the address of INT 0xE in the IDT.

Now start Tuluka up and scan the system again with GMER.

Now also start Tuluka up and see if it shows itself as having hooked that specific interrupt.

Nope. Not at all. A rootkit detector exhibiting rootkit behavior. Interesting to say the least, if you ask me. It turns out though (after me reading plenty of literature on the subject) that it is not uncommon to find that detectors have also hooked the system. I'll leave it to you, on what software to trust and what not to – all I'm showing you here is an example of a hooked IDT.

SYSENTER Hooks

Just a while ago, I mentioned that the IDT is used only in older systems to jump into Kernel mode. In WinXP and beyond, when a function in kernel mode needs to be called the processor fires an instruction called SYSENTER. This looks inside a register named IA32_SYSENTER_EIP

to get the address of the code which handles the SystemServiceTable, and then jump to that code. A rootkit would save the address in IA32_SYSENTER_EIP and overwrite it with its own address. So every time SYSENTER is executed, it'll jump to the rootkit's code which handles all the system service calls; instead of the original table. Potentially, a rootkit could have an entire new service handling table of its own somewhere, with every single function in the service table hooked as well; so everything is always hooked!

Tuluka and GMER both detect SYSENTER hooks. In Tuluka you can find it under the Sysenter tab. In GMER you need to go to the Rootkit/Malware Tab and tick System and scan. Tick the 'Show All' box. Now look for an entry which starts with SYSENTER.

An interesting thing here is that at any time, there will always be only 1 SYSENTER entry in any rootkit detector. This is because it is always the IA32_SYSENTER_EIP that is looked into. Just one. This is unlike the IDT, where there is a line for every interrupt.

Lets look at an example of a rootkit which hooks SYSENTER. Here is a screenshot of Tuluka before I ran the rootkit. Note the address of the handler.

Now lets look at both Tuluka and GMER after I ran the rootkit. Look at the address in both places. The funny part though is that it doesn't say that it is suspicious, despite a change happening. GMER does so though.

SSDT Hooks

Whenever there is a switch to kernel mode that happens, an instruction called SYSENTER is executed transparent to the user; signifying that kernel mode execution has to happen and we want to run a system service (execute a kernel mode API). To do this the address of the SystemServiceDispatcher needs to first be found. Remember that the user mode API had the number of the kernel mode API? The SystemServiceDispatcher uses that number to look into the SSDT, which contains a complete System Service Number --->KernelModeAPIaddress mapping. It gets the address of the API and then runs it. In the case of FindFirstFile, the kernel mode API is NtQueryDirectoryFile ;this runs and displays the results to the user.

You might have guessed what the rootkit will do here. If you did, well done. If you didn't, no problem – that's where this article comes in :). The rootkit will look at the SSDT and look at the mapping for NtQueryDirectoryFile. It'll keep the service number intact but change the address of the function to the rootkit's own function instead. So the next time FindFirstFile is called, it'll lookup the address of NtQueryDirectoryFile using the service number. Due to Mr.Rootkit though, it will be taken to the rootkit's function instead of the original NtQueryDirectoryFile. The rootkit eventually will call the original NtQueryDirectoryFile, manipulate its results and return the manipulated stuff to the user.

Tuluka as well as GMER detect SSDT hooks. In Tuluka the results of an SSDT scan are available under the SST Tab. In GMER you navigate to the Rootkit/Malware Tab and check only System on the right hand side. You can tick the ShowAll box below in GMER, if you want to see all valid entries as well.

We'll look at a very interesting example now. I'm sure a lot of the malware analysts out there would have used a tool called ProcMon to monitor file I/O and registry activity. How does ProcMon monitor all this? As it turns out it hooks the SSDT for part of its monitoring anyway :). Lets look at a small series of screenshots to prove this.

Here is a Tuluka scan before ProcMon was ever started on this machine. Focus on the NtCreateKey [just as a sample]

Now I started up another monitoring tool called CaptureBat to see what ProcMon actually does. Have a look at the snipped CaptureBat output in the next screenshot.

Notice that Procmon is writing a file PROCMON20.SYS into System32? Very interesting. And then deleting it as well.....

And then we run Tuluka again...

Red Line. Changed address. SSDT Hook. The even cleaner part is that you can no longer see the file in System32; that was deleted as you saw in the previous screenshot.

Again..I leave it to you what you want to trust. Having said that though, ProcMon is a tool you can trust, so you don't really need to be worried if you've downloaded it from the official site. If you're paranoid though; Tuluka offers you a nice feature to right click on that line and clear the hook. Right click on Red Line – Restore Service. Check the addresses. Yep..all okay. Here's a screenshot after restoring.

Hooking specific APIs in the kernel..

We briefly mentioned in our User hooks section, that we could hook in Kernel mode. Lets look at that in detail now, as we have the necessary background to do so. So our normal process till looking up the address of a kernel API in the SSDT will be exactly the same. No tampering anywhere.

If you remember, the kernel API that FindFirstFile calls is NtQueryDirectoryFile. The rootkit will get the address of this function from the SSDT and hook it. We have not hooked the SSDT this time; just 1 single API in the SSDT. So each time FindFirstFile runs the fake NtQueryDirectoryFile runs.

The rootkit code will then call the original NtQueryDirectoryFile and wait for it to return. It will then read the name of the file and check if the filename is the component it wants to hide. If it is, then it will remove the names of its own files, before displaying the result to the user. So, unlike in UserMode, even if you use a third party tool, you can't see the rootkit, as the tool also uses the same APIs (FindFirstFile and FindNextFile) to display a file listing. As you might have guessed, rootkit developers need to hook every possible API that can be used to display a list of files in a directory.

Detection mechanisms here are exactly similar to those found in the User Mode rootkit section. Tuluka doesn't have anything explicit, while GMER does under the Files tab.

Here's a screenshot where we check if any CODE sections have been modified by checking just the Sections box in GMER.

Now look at a GMER screenshot after a rootkit was installed.

Hiding drivers..

A driver is a piece of software that enables applications to interact with hardware. So for example if you try and read a file from a drive, there'll be multiple drivers that are invoked from the time you use the mouse to double click a file to the actual reading of the raw data from the sectors on the hard disk. Your input is handed off to Driver 1 who processes it and hands it to Driver 2 and so on till it reaches the driver that deals with reading raw data from the hard disk. So whenever you start up your operating system all the necessary drivers are loaded into memory and ready to function when called upon. So for example: There will be a driver to handle USB devices which is activated when you plug a USB drive in.

So to sum up, a driver will have plenty of different functions [Read,Write, Close, Unload etc], each of them called when there is some need for either an application or some other component. To deal with each of these, the driver will have many subroutines in its code, which are called when needed.

Lets try and understand how a driver deals with these requests. Yes, I know this is a little too heavy but bear with me – I'll try and simplify it as much as I can. Every driver's first bit of code is to call a function called DriverEntry which takes 2 arguments – a structure called DRIVER_OBJECT and a path to the Registry where further driver specific info is stored. In a nutshell the DriverEntry routine fills up the DRIVER_OBJECT structure with other entry points to the driver for handling each relevant function [Read,Write etc]. So for example: 1 sample line of code could look like this:-

pDriverObject->MajorFunction[IRP_MJ_READ] = Example_Read;

This just means – whenever "something" wants to read and this driver is needed, send the request to the function Example_Read; it'll deal with it.

These requests are all called IRP Major requests; there are numerous such lines that you'd have to write , depending on which functions your driver is handling. The function Example_Read is called an IRP handler function...and it is these functions that a "driver" rootkit will look to hook. Recollect the FindFirstFile example.... A call to the FindFirstFile API first jumped to the rootkit, the rootkit then called FindFirstFile(original), the rootkit waited for FindFirstFile to return its list of files, edited this result and then displayed it to the user. The logic for hooking IRP handlers also is very similar; the rootkit function will hook IRP_MJ_READ, then call Example_Read and manipulate its returned results.

Tuluka as well as GMER have features to scan for modified or hooked drivers. The results for these can be found in the Drivers and Devices tab in Tuluka while ticking only Drivers in the Rootkit/Malware tab in GMER will give you a list of hooked drivers.

Highlighting an entry in Tuluka's list of displayed drivers (Drivers Tab), gives you details about the IRP handlers and their addresses in the middle tab and even the disassembly in the lowermost pane in Tuluka. In GMER you have to go to the Rootkit/Malware section; rightclick anywhere and select Options -> IRP Hooks and then rescan.

Here is an example of a rootkit that hooks a driver or adds a driver of its own. Here is a screenshot of the TcpIp driver before the rootkit was installed.

Now have a look at both Tuluka and GMER after the rootkit was run.

Call Gate/GDT Hook detection...

Code in UserMode needs to call code in KernelMode all the time. We've already seen one way in which this is done; using the INT 0x2E or the SYSENTER instruction. Another way this is done is by using something called Call Gates.

Think of a CallGate as a regulating mechanism. I mean, that's what gates are for ..right? To keep unwelcome intruders out? A CallGate does exactly that, it allows user mode applications to call code in KernelMode but regulates who can call what code.

A CallGate is stored inside a table in memory called the GlobalDescriptorTable(GDT). Don't worry about the fact that you've never heard of it; I hadn't either at one time ;). In this context all you need to know is, that programs can use CallGates only if they know its index or position in the GDT. So an app program simply looks into the GDT, gets the index of the CallGate and calls the CallGateDescriptor at that index.

The CallGate does a load of fancy things which are too detailed to go into in this article. In a nutshell though, here are the major features of a CallGate:

• Is the CallGateDescriptor even valid?
• Where in the kernel is the privileged code present?
• What privilege level does the caller need, so he can call this code?
• Does the caller need to pass any paramters? If so what are those?

If all of those conditions (including others which I haven't mentioned) are met satisfactorily, the user mode program is allowed to talk to the kernel mode code and run it. So now that we have a high level understanding of what a CallGate does..try and guess how a rootkit would try and exploit this?

Well, it would just create a new CallGate and add it to an empty slot in the GDT. The CallGate would have an index which the rootkit could call; it'd also have the details of the KernelMode API you want to sneakily call from user mode. So once the CallGate's configured correctly, the rootkit can execute the targeted function.

Tuluka has a feature where it scans for suspicious items in the GDT (sneaky CallGate) but GMER does not seem to have this feature.

Let us take a look at how Tuluka deals with a call gate. Here is a screenshot of what a scan looks like before a rootkit that installs a call gate is created.

Now I installed a POC which implements a call gate and ran a scan again; Tuluka detected that a call gate was indeed present.

Other features....

Apart from all of these, which I felt were the most signficant features, Tuluka has other features where you can view a list of all SystemThreads as well as the content of all debug registers or system memory. You can suspend specific threads, kill specific processes or restore a tampered entry in the SSDT if you want to. I won't go too much into detail of any of these or its other features. I'd strongly advise you download both Tuluka and Gmer and play with them yourself to get a feel of what it can or cannot do.

Rootkits that I used...

I used a few different rootkits for all my examples in this article. In some places I just downloaded and installed POCs that other authors had written; as the purpose was to just show how the rootkit detector actually detects modifications made to the system.

A few rootkits/detectors/monitoring tools that I used while testing were HackerDefender, PE386, AFX, FU, Mailbot.AZ/Rustock, ShadowWalker, CallGate POC from the Rootkit Arsenal book [modified by someone and put online], ProcMon and Tuluka itself in one case ;). I tried out a whole lot more to be frank; but these were the ones whose results I used in the article itself.

Conclusion..

Well, I hope you guys enjoyed that introduction to all the bad things that a rootkit does to hide itself. I have focussed just on introducing you to all of these features in this article and deliberately not made an attempt to dive deeper into how exactly the rootkit detectors work.

Maybe, in a future article I'll look at how exactly a rootkit detector actually identifies that something's hooked. How do some rootkits even offer to remove that hook? Are there other techniques apart from hooking that a rootkit can use to hide itself, but cause a similar amount of damage? How can I write a rootkit itself? Lots of very interesting questions; maybe someday I'll answer all of them. Thanks for reading :)

References:

Books:

Learn Digital Forensics

Build your skills with hands-on forensics training for computers, mobile devices, networks and more.

Start Learning

Learn Digital Forensics

Build your skills with hands-on forensics training for computers, mobile devices, networks and more.

Start Learning

• The Rootkit Arsenal - http://books.google.com/books?id=5Cs46M9FV0sC&lpg=PP1&dq=bill%20blunden%20rootkit%20arsenal&pg=PP1#v=onepage&q&f=false
• Rootkits: Subverting the Windows Kernel - http://books.google.com/books?id=fDxg1W3eT2gC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false

Other excellent articles, presentations and PDF documents

• Tuluka – www.tuluka.org
• GMER -- http://www.gmer.net/
• Good introduction to Hooking - http://blog.fireeye.com/research/2008/03/rootkits---ma-1.html
• High level view of rootkit techniques - http://www.security-assessment.com/files/documents/presentations/2006-04_DB_Rootkits_Advanced%20Malware%20-%20Brightstar_April06_DB_190406.pdf
• Another High level view - http://crackthecode.us/windows_rootkits/Operating_System_Security.ppt
• Super Paper (Detailed) – Windows Rootkits - http://web.archive.org/web/20061015221852/http://vigilantminds.com/files/inside_windows_rootkits.pdf
• Introduction to Driver design - http://www.codeproject.com/KB/system/driverdev.aspx
• Driver to hide processes and files - http://www.codeproject.com/KB/system/hide-driver.aspx
• Call Gate - http://en.wikipedia.org/wiki/Call_gate
• Intel Manual Vol 3A – Section 5.8.3 – Call Gates
• Superb training series [Do support if you find this useful] - http://opensecuritytraining.info/Rootkits.html