Malware analysis

Anti-disassembly, anti-debugging and anti-VM

Malware takes advantage of a disassembler’s basic assumptions. It exploits predictable choices made by debuggers and hides when it suspects it is running on a virtual machine. 

This article offers an overview of malware defensive tricks and techniques used when disassembly and debugging tools are searching for malware, especially within a virtual machine. 

Anti-disassembly techniques

Malware authors use anti-disassembly techniques to delay, prevent and/or avoid the reverse-engineering of their code. It uses manually crafted code to cause disassembly analysis tools to produce an incorrect program listing. Here are some common anti-disassembly techniques.

API obfuscation

API obfuscation changes the names of identifies (class names, method names, field names) to random names so that the reader of the code doesn’t know what the code is doing. 

Opcode/assembly code obfuscation

Opcode/assembly code obfuscation makes disassembly of malware difficult by using tactics like executables with decrypted sections and code instructions that are hard to read or nonsensical. 

Junk/spaghetti code

Junk/spaghetti code is used to confuse the reverse engineer and hide what the current code is trying to accomplish. 

Control flow graph flattening

Control flow graph flattening, or simply CFG flattening, flattens the control flow of each function by first breaking up the nesting of loops and if-statements and then hiding each of them in a case of a large switch statement wrapped inside the body of a loop.

Jump instruction with same target

Jump instruction with the same target is produced using a combination of jz with jnz. This is an unconditional jump that the disassembler doesn’t recognize because it only disassembles one instruction at a time.

Detecting debuggers

Anti-debugging is an anti-analysis technique that is used by malware to check if it is being debugged. Malware authors use many techniques to prevent and or slow the reverse engineer from debugging their code. 

Windows API

The most common technique malware uses is Windows API, as it provides several functions that can be used by malware to detect debuggers. The most used one is IsDebuggerPresent. It checks for a specific flag in the Process Environment Block (PEB) for the field BeingDebugged, which will return zero if the process is not running into a debugger or nonzero if a debugger is attached. 

Another similar function is CheckRemoteDebuggerPresent, which checks if a remote process is debugging the current one. 

CloseHandle/NtClose

CloseHandle/NtClose is another anti-debugging malware uses. Calling the dispatcher with an invalid handle throws an invalid handle exception, STATUS_INVALID_HANDLE. 

FindWindow

FindWindow is also used to find the debugger by providing window class (e.g., OLLYDBG).

NtGlobalFlag

Malware can also take advantage of PEB’s NtGlobalFlag flag at offset 0x68 (which is called to check if it’s being debugged) by verifying if its value is equal to 0x70.

Advanced anti-debugging techniques

Debugging specialists and analysts who encounter the following advanced anti-debugging techniques will find themselves and their tools challenged. 

Timing defense 

The Read Time-Stamp Counter (RDTSC) is a common example of a timing defense. Debugging tools take time and effort to process each instruction. This process creates slowness and fluctuation that is different from the computer’s fixed processing speed threshold. Malware avoids detection by reading the computer’s time stamp multiple times; if it finds time irregularities, it shows an error because it knows a debugger is searching the system.

Breakpoint detection

There are various types of breakpoints, but the following are most commonly used by reverse engineers. 

• Software breakpoints are set by replacing the instruction at the target address with the byte 0xCC (INT3) so the malware can count the number of 0xCC bytes (for example, using the instruction repne scasb) and to determine whether the program is being debugged or not. Another way that malware uses is by performing a CRC or MD5 checksum of the opcodes

• Hardware breakpoints are detected by using GetThreadContext/SetThreadContext APIs and checking if DRs are set. If yes, then hardware breakpoint is set

Interrupts

An interrupt is a condition that halts the microprocessor temporarily or permanently to work on a different task called an interrupt routine (also called Interrupt Service Routine or simple ISR) whose address is stored in the Interrupt Descriptor Table (IDT). It then returns to its previous task. Malware can use some interrupt instructions to detect a debugger. The following are some of these instructions:

• Two-byte INT3: (0xCC, 0xCD+0x03) Can be used as false breakpoint
• INT 0x2C: Raises a debug assertion exception
• INT 0x2D: Issues an EXCEPTION_BREAKPOINT if no debugger is attached
• ICEBP (0xF1): Generates a single-step exception
• Trap flag: Generates a single-step exception (int 0x01h) after executing an instruction
• Stack segment: Tracing over SS (e.g., mov ss, pop ss), the debugger will not break on those, effectively stopping on the following instruction. In other words, an unset of the trap flag won’t be possible after that, and if check is done here, a debugger will be detected

TLS callbacks

Almost all debuggers start the program’s entry point, as defined by the PE header. Thread Local Storage (TLS) callback is a technique that can be used to execute a code before or after the main application code execution. This means that when loaded in a debugger, the program does all the checking before reaching the entry point and the analyst wouldn’t know what is happening.

Anti-VM techniques

Malware authors know that most malware analysts prefer to run their malware inside a virtual machine environment to avoid their computers being affected by the malware they’re trying to analyze. That is why most malware authors use anti-VM techniques to make it harder to see or analyze their code within a virtual machine. Here are some of these techniques:

CPUID 

This is the most popular among these techniques. This instruction is executed with EAX=0x1 as input, and the return value describes the processors features. The 31st bit of ECX on a physical machine will be equal to 0. On a guest VM, it will equal to 1. 

Another method using CPUID executes it with EAX=0x40000000, which is called “hypervisor brand.” The return result will be the virtualization vendor (“VMwareVMware” for VMWare and “Microsoft HV” for Microsoft). The result is stored in ECX and EDX.

Other methods that malware use are Checking for Known Mac Addresses (for VMWare it’s 00:05:69, 00:0C:29, 00:1C:14, 00:50:56, for VirtualBox it’s 08:00:27), Checking for Registry Keys, Checking for Processes, Checking for Files or Checking for Running Services.

Red Pill and No Pill

Red Pill is an anti-VM technique that executes the SIDT instruction to grab the value of the IDTR register. The VM monitor must relocate the guest's IDTR to avoid conflict with the host's IDTR. Since the VM monitor is not notified when the VM runs the SIDT instruction, the IDTR for the VM is returned.

No Pill relies on the fact that LDT structure is assigned to a processor, not an OS. I’s location on a host machine will be zero and on a VM will be non-zero.

Conclusion

Malware uses guerrilla strategies, rewriting the rules of engagement. It avoids discovery by reordering processes, arriving early or late, recognizing a trap and obfuscating the tools that seek to disassemble and debug it. 

Malware, however, is limited by its programming and relies on predictability. Adaptive malware analysts are able to adjust their strategy in real-time when they recognize these malware defensive techniques, counteracting them. 

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

Sources

• The "Ultimate" Anti-Debugging Reference, Peter Ferrie
• The Art of Unpacking, Mark Vincent Yason
• Secrets of Reverse Engineering, Eldad Eilam
• Anti Reverse Engineering Techniques Database, OpenRCE
• IsDebuggerPresent function, Microsoft 
• CheckRemoteDebuggerPresent function, Microsoft
• Anti Debugging Tricks #2 – CloseHandle, Game Phreakers
• PEB-Process-Environment-Block/NtGlobalFlag, Aldeid