VMPmonitor: An Efficient Modularity Approach for Hidden Process Detection

Chaoyuan Cui, Wu Yun, Li Ping, Rujing WangAdvances 2013 · DOI

Abstract

With the development of Cloud computing, more and more people are accustomed to resource sharing or online shopping. And malware has become a major threat to the Cloud safety. Process hiding is a powerful technique commonly used by stealthy malware to evade detection by anti-malware.

Introduction

VMPmonitor is a lightweight, Xen-based security tool designed to unmask "hidden" processes—malicious software that conceals itself from a guest operating system's standard monitoring tools. By analyzing the guest's memory and CPU registers from the safety of the hypervisor (the "outside"), VMPmonitor provides a reliable, tamper-resistant view of what is actually running on a virtual machine.

The Problem: Stealthy Malware in the Cloud

In cloud computing environments, malware—specifically rootkits—often uses a technique called process hiding. By modifying the guest operating system's internal structures or intercepting system calls, a rootkit can make its malicious processes invisible to standard administrative tools like ps or top.

If a process is invisible, it cannot be analyzed or terminated. Traditional anti-malware running inside the virtual machine (VM) is vulnerable because the malware can simply lie to the security software. VMPmonitor addresses the need for "out-of-the-box" monitoring: a way to inspect a VM's activity from the hypervisor level, where the malware cannot easily interfere with the detection mechanism.

Methodology

The authors propose a modular architecture called VMPmonitor, built on the Xen Hypervisor. Its core innovation lies in how it locates process information without relying on the guest OS's potentially compromised APIs.

Register-Based Locating and Kernel Traversal

Instead of asking the guest OS for a process list, VMPmonitor looks at the guest's hardware state. Specifically, it uses the ESP (Stack Pointer) register. In Linux, every process is represented by a task_struct. By using the ESP register and the known THREAD_SIZE of the kernel, VMPmonitor can calculate the address of the thread_info structure, which contains a pointer to the task_struct.

Mapping the ESP register to the task_struct via thread_info.

Once one task_struct is found, VMPmonitor follows the kernel's internal doubly linked list of processes. Because this list is fundamental to how the CPU schedules tasks, it is much harder for a rootkit to manipulate this list without crashing the entire system.

Experimental Validation

The paper validates the tool using a controlled experimental environment:

  • Hypervisor: Xen 4.1.2 running on Ubuntu 12.04.
  • Guest OS (DomU): CentOS 6.4 (64-bit) with Linux kernel 2.6.32.
  • Malware Sample: The adore-ng rootkit, a well-known advanced Linux kernel rootkit used to hide files and processes.

Key Results

The authors conducted a "hidden process" experiment to prove the tool's efficacy. They used the adore-ng rootkit to hide a specific process (sshd with PID 1202) within the CentOS guest.

VMPmonitor detecting the hidden sshd process (PID 1202) while the guest OS fails to list it.

As shown in the experimental results, the standard ps -e command inside the guest VM failed to show the sshd process; the rootkit successfully "cloaked" it from the internal OS view. However, VMPmonitor, running from the hypervisor, successfully identified the hidden process and its PID.

Unlike previous external monitors like Lycosid, which use complex statistical analysis and can suffer from false positives, VMPmonitor is deterministic and lightweight. Because it performs direct memory inspection with minimal data post-processing, it does not significantly impact the performance of the guest VM.

Conclusion and Future Work

VMPmonitor demonstrates that implicit information—like hardware registers—provides a "ground truth" that guest-level malware cannot easily spoof. While currently focused on Linux processes, the authors plan to extend this modular approach to detect hidden files, network ports, and registry keys across diverse operating systems.

Summary updated

Abstract

With the development of the Cloud computing, more and more people are accustomed to resource sharing or online shopping. And malware has become a major threat to the Cloud safety. Process hiding is a powerful technique commonly used by stealthy malware to evade detection by anti-malware.
BibTeX
@inproceedings{cui2013vmpmonitor,
  author = {Chaoyuan Cui and Wu Yun and Li Ping and Rujing Wang},
  title = {{VMPmonitor: An Efficient Modularity Approach for Hidden Process Detection}},
  booktitle = {Advances},
  year = {2013},
  doi = {10.2991/ccis-13.2013.59},
}

Loading related content...

Figure