Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction

Xuxian Jiang, Xinyuan Wang, Dongyan XuACM Transactions on Information and System Security 2010 · DOI

TL;DR

This paper introduces VMwatcher, a novel system that bridges the "semantic gap" in virtual machine (VM) introspection for malware detection. By reconstructing a VM's internal semantic view from an "out-of-the-box" perspective, VMwatcher enables robust, tamper-resistant malware detection and monitoring without requiring agents within the VM.

What Problem Does This Paper Address?

Traditional host-based antimalware systems run inside the very systems they protect ("in-the-box"). This grants them a rich, semantic understanding of the system's state (files, processes, etc.) but makes them vulnerable to detection and subversion by sophisticated malware. To overcome this, researchers have proposed "out-of-the-box" solutions that run outside the VM, offering tamper resistance due to VM isolation. However, this approach suffers from a "semantic gap": the external observer only sees low-level data (memory pages, registers) and lacks the high-level understanding of system objects that "in-the-box" systems enjoy. This makes it difficult to detect complex malware behaviors. VMwatcher aims to provide the best of both worlds: the tamper resistance of an external observer with the semantic insight of an internal one.

Methods

VMwatcher employs two core techniques:

  • Nonintrusive Virtual Machine Introspection (VMI): This involves passively observing low-level VM states and events (registers, memory, disk, system calls) directly from the Virtual Machine Monitor (VMM) without installing any software inside the guest VM. This ensures that VMwatcher itself doesn't perturb the VM's execution or become a target for malware.
  • Guest View Casting: This is the novel technique that bridges the semantic gap. It works by "casting" the semantic definitions of guest OS data structures and functions (like those for files, processes, and kernel modules) onto the low-level VM states captured by VMI. Essentially, VMwatcher uses the guest OS's own understanding of its internal workings as a template to interpret the raw data it observes from the outside. This allows for the reconstruction of high-level semantic views, such as lists of files, running processes, and even detailed system call information (including the caller, call number, and parameters).

VMwatcher extends this to reconstruct system call events, providing detailed information about which process made the call, the call itself, and its arguments and return values.

Datasets

The paper does not explicitly mention training datasets. Instead, it focuses on the evaluation of VMwatcher using:

  • Real-world malware: A collection of stealthy rootkits (both Windows and Linux) and worms were used to test VMwatcher's detection capabilities.
  • Off-the-shelf antimalware software: Various antivirus products (Symantec AntiVirus, Microsoft Windows Defender, Kaspersky, etc.) were deployed both internally and externally (via VMwatcher) to compare detection effectiveness.
  • Standard benchmarks: UnixBench and ApacheBench were used to measure the performance overhead of VMwatcher's system call monitoring.

Key Insights and Ideas

  • Bridging the Semantic Gap is Crucial: The core insight is that simply observing low-level VM states isn't enough for effective malware detection. Reconstructing a semantic view is essential, and guest view casting provides a systematic way to achieve this.
  • "Out-of-the-Box" with Internal Insight: VMwatcher demonstrates that it's possible to gain the tamper resistance of external monitoring without sacrificing the detailed understanding of system internals.
  • View Comparison for Stealthy Malware: A key capability is comparing an internal view (if available) with VMwatcher's external semantic view. Discrepancies can reveal hidden processes or files, a common tactic for rootkits.
  • Cross-Platform Antimalware: VMwatcher enables running antimalware designed for one OS (e.g., Windows) to scan a VM running a different OS (e.g., Linux), opening up new possibilities for malware analysis.
  • Nonintrusive System Call Monitoring: This capability is invaluable for understanding malware behavior in a controlled environment without alerting the malware itself.

Key Results

  • Rootkit Detection: VMwatcher successfully detected various stealthy rootkits (e.g., FU rootkit, Hacker Defender, NTRootkit, adore-ng) by comparing internal and external views, exposing hidden processes and files that internal tools missed.
  • Antimalware Performance: In performance comparisons, external scanning using VMwatcher was often faster than internal scanning by antimalware software, potentially due to reduced disk I/O overhead and efficient host-side processing.
  • System Call Monitoring Overhead: The overhead for real-time system call interception and reconstruction was generally low, with UnixBench showing worst-case overheads of 11.0% and ApacheBench showing a 3.8% degradation in throughput.
  • Memory Analysis Speed: Analyzing raw Windows memory images was slower than Linux due to the lack of readily available kernel symbols, but Linux memory analysis was very fast (under 0.5 seconds regardless of VM size).

Limitations and Future Work

  • Trusted VMM Assumption: VMwatcher assumes the VMM itself is trusted and not compromised. Attacks that subvert the VMM are outside the scope of this work.
  • Guest View Subversion: Malware could attempt to subvert the guest OS's internal functions or data structures to mislead VMwatcher's semantic reconstruction. The authors suggest extending VMwatcher to validate the integrity of guest kernel objects.
  • Guest Caching Exploitation: Malware might hide in disk caches, avoiding detection if changes aren't immediately reflected. The authors suggest examining cached contents directly in memory.
  • VM Fingerprinting: Attackers can detect if they are running in a VM. While VMwatcher aims to mitigate this by improving VM fidelity, more fundamental detection methods are challenging.
  • Future Work: The authors suggest further research into secure booting and trusted VMMs to bolster the foundation of their approach. They also highlight the need to handle variations in OS configurations and kernel data structures for broader applicability.

Summary updated

Abstract

An alarming trend in recent malware incidents is that they are armed with stealthy techniques to detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based antimalware systems is that they run inside the very hosts they are protecting (“in-the-box”), making them vulnerable to counter detection and subversion by malware. To address this limitation, recent solutions based on virtual machine (VM) technologies advocate placing the malware detection facilities outside of the protected VM (“out-of-the-box”). However, they gain tamper resistance at the cost of losing the internal semantic view of the host, which is enjoyed by “in-the-box” approaches. This poses a technical challenge known as the semantic gap. In this article, we present the design, implementation, and evaluation of VMwatcher —an “out-of-the-box” approach that overcomes the semantic gap challenge. A new technique called guest view casting is developed to reconstruct internal semantic views (e.g., files, processes, and kernel modules) of a VM nonintrusively from the outside. More specifically, the new technique casts semantic definitions of guest OS data structures and functions on virtual machine monitor (VMM)-level VM states, so that the semantic view can be reconstructed. Furthermore, we extend guest view casting to reconstruct details of system call events (e.g., the process that makes the system call as well as the system call number, parameters, and return value) in the VM, enriching the semantic view. With the semantic gap effectively narrowed, we identify three unique malware detection and monitoring capabilities: (i) view comparison-based malware detection and its demonstration in rootkit detection; (ii) “out-of-the-box” deployment of off-the-shelf anti malware software with improved detection accuracy and tamper-resistance; and (iii) nonintrusive system call monitoring for malware and intrusion behavior observation. We have implemented a proof-of-concept VMwatcher prototype on a number of VMM platforms. Our evaluation experiments with real-world malware, including elusive kernel-level rootkits, demonstrate VMwatcher's practicality and effectiveness.
BibTeX
@article{jiang2010stealthy,
  author = {Xuxian Jiang and Xinyuan Wang and Dongyan Xu},
  title = {{Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction}},
  journal = {ACM Transactions on Information and System Security},
  year = {2010},
  doi = {10.1145/1698750.1698752},
}
Figure