Malware analysis

Virtual Machine Introspection in Malware Analysis – API Monitoring

March 5, 2019 by Youness Zougar

LibVMI allows you to manipulate the memory, but in any case, it allows you to directly analyze malware behavior. In order to take advantage of LibVMI’s introspection system, it will be necessary for us to develop our own system to analyze malware based on the APIs provided by LibVMI.

In this article, we will focus on the monitoring of Windows Kernel APIs in order to trace the behavior of the malware. To determine the APIs to be monitored, it will be necessary to use the Rekall (Forensics framework) feature. This will allow us to download the .pdb file corresponding to the version of the Windows Kernel of the virtual machine, with the aim of recovering the offsets corresponding to the Kernel API. Finally, it will create a JSON file containing the names of the APIs or Windows structures and the corresponding offsets.

Below is an extract from the generated JSON file:


"NtCreateEnlistment": 3100512,

"NtCreateEvent": 3403920,

"NtCreateEventPair": 4530944,

"NtCreateFile": 3662112,


To the offsets, we add the Kernel base address to find the addresses in the memory of the Kernel APIs.

LibVMI provides different ways to monitor and interact from the hypervisor with the virtual machine memory.

Monitoring via Breakpoints

The idea is here to place breakpoints (often used in reverse-engineering) on ​​the APIs that we want to monitor in the virtual machine during the initialization of the introspection system, while maintaining a lookup table of offsets of breakpoints placed and bytes modified. After the installation and launch of the introspection system, the malware calling the APIs on which we have placed breakpoints will be interrupted and a notification will be sent back to the hypervisor allowing us to find which executable has reached the breakpoint. Once the desired information (function name, its arguments) is recovered, we continue the execution of the malware (not changing its behavior) until the next breakpoint. Thanks to this information, we can understand the behavior of the analyzed malware.

The use of this technique alone has two weaknesses. The first is in the situation where the malware makes a checksum on itself to detect if its code has been altered. If this is the case (since breakpoints modify the content), the malware may stop executing to not reveal its malicious behavior. The second is in the situation where we want to monitor Kernel APIs. Here, it is possible to cause a BSOD in Windows, since Windows performs an integrity check of the Kernel and may crash if it has ever been modified.

Monitoring Via Memory Pages

With this technique, there is no need to place breakpoints and thus alter the memory. The idea would be to place a monitoring with memory events on each executable memory page and to calculate the address of the APIs that we wish to monitor. Whenever a region of the page marked in execution is accessed, the information is returned, and a comparison is made with the address of the monitored API.

This technique solves the problem of altering the memory with breakpoints but introduces others. Monitoring on each executable page would be expensive in terms of resources and would cause a considerable slowdown of the virtual machine. Do not forget that the virtual machine will not have a lot of resources to work, especially if you want to automate the analysis on multiple virtual machines in parallel.

Monitoring via Breakpoints + altp2m

As described above, using breakpoints alone will not work, so Xen’s developers have been thinking of a technique called altp2m. This technique aims to create a shadow copy of the desired memory pages. The idea is to create an altered view (containing breakpoints on Kernel APIs that we want to monitor) and keep an intact view that will be returned each time there is read or write access to these memory pages.

Thanks to this method, we solve the two problems mentioned in the Monitoring via Breakpoints chapter.

Thank you for following this new part of VMI applied to malware analysis. In the next part, we will develop our own system of malware analysis based on LibVMI functions. We will see how to determine interesting Kernel APIs to monitor and how to get valued information from them.

Posted: March 5, 2019
Youness Zougar
View Profile

Youness Zougar is an IT Security Expert, especially in Reverse Engineering and Malware Analysis. He is also a Windows Kernel Drivers developer and co-founder at SAFE-Cyberdefense. In order to offer companies the best level of security, he developed with his teammates an Endpoint solution called SAFE Endpoint Security based on a defense in depth technology applied to workstations.