Protecting Critical Files Using Target-Based Virtual Machine Introspection Approach

Dongyang Zhan, Lin Ye, Binxing Fang, Xiaojiang Du, Zhikai XuIEICE Trans. Information and Systems 2017 · DOI

Introduction

The protection of critical system files, such as /etc/passwd in Linux or system drivers in Windows, is a fundamental requirement for operating system security. Traditional security monitors typically operate as agents within the guest operating system. However, this internal placement creates a significant vulnerability: if a rootkit compromises the kernel, it can easily disable or bypass these security agents.

Virtual Machine Introspection (VMI) addresses this by relocating the monitor to a separate, higher-privileged "Secure VM" or the hypervisor itself. While VMI provides isolation and stealth, existing "operation-based" VMI tools often suffer from prohibitive performance overhead. These tools intercept every file-related system call (e.g., open, read, write) to determine if a protected file is being accessed. In active production environments where thousands of file operations occur every second, this approach is often impractical.

System Architecture

This paper introduces CFWatcher, a security tool designed to protect critical files with significantly lower overhead by shifting from operation-based monitoring to a "target-based" approach.

The CFWatcher architecture, showing the separation between the Secure VM and the Target VM via the Virtual Machine Monitor (VMM).

As shown in Figure 1, the CFWatcher architecture consists of a Secure VM containing a Policy Engine and a Toolkit Library. This Secure VM interacts with the Target VM through a VMI interface provided by the Virtual Machine Monitor (VMM), specifically the Xen hypervisor. This design ensures that the security logic is completely isolated from potential compromises within the Target VM's Guest OS.

Methodology: Target-Based Monitoring

The core innovation of CFWatcher is its focus on metadata objects rather than system actions. Instead of monitoring every system call, CFWatcher tracks the specific memory structures that represent files in RAM.

  1. Metadata Tracking: In Linux environments, CFWatcher monitors dentry (directory entry) objects. In Windows, it tracks _SCB (Stream Control Block) objects. These structures are loaded into memory whenever a file is accessed.
  2. Hardware-Assisted Triggers: CFWatcher utilizes Intel’s Extended Page Tables (EPT) to enforce protection. It marks the memory pages containing the metadata of target files as read-only. Any attempt to modify these files triggers an EPT violation, which the hypervisor catches and redirects to the CFWatcher policy engine for analysis.
  3. Bridging the Semantic Gap: To translate raw binary memory into high-level OS concepts like filenames and process IDs, CFWatcher integrates the Volatility Framework.

Cache Liveness and System Call Injection

A challenge with target-based monitoring is that metadata must remain in the system cache to be monitored. If an object is evicted from memory, the EPT protection is lost. CFWatcher addresses this through "cache liveness" strategies.

Comparison of standard system call execution versus the CFWatcher injection method used to maintain metadata in memory.

As illustrated in Figure 6, the authors propose a system-call-based approach where an open call is injected into the kernel. This ensures that the target file's metadata is loaded and kept "warm" in the cache. Alternatively, an agent-based approach can be used where a small, stealthy daemon opens the files at boot time.

Evaluation and Results

The authors evaluated a prototype of CFWatcher built on the Xen hypervisor using Ubuntu 12.04 and Windows 7 guest environments. Performance was measured using standard benchmarks like dd, kernel unzipping, and PCMark 7.

  • Efficiency: During a stress test involving unzipping the Linux kernel—a task characterized by high-frequency small file creation—CFWatcher introduced only 19.2% overhead. In contrast, a traditional operation-based VMI approach resulted in a 212% overhead.
  • Scalability: In Linux, protecting 50 files resulted in a performance loss of less than 5%. On Windows, protecting 50 files simultaneously led to a negligible 1.6% performance drop in PCMark 7.
  • Security Effectiveness: CFWatcher successfully detected and blocked the Phalanx rootkit's attempts to hijack system calls and prevented unauthorized modifications to the Windows Host file.

Conclusion and Future Work

CFWatcher demonstrates that target-based VMI can provide robust, stealthy file protection without the massive performance penalties of traditional methods. By protecting the underlying memory structures, the tool implicitly monitors all access paths to a file.

However, some limitations remain. Performance can be impacted by "memory page density," where an unprotected, frequently accessed file shares a memory page with a protected metadata object, leading to false EPT triggers. Future work will focus on optimizing page alignment and evolving CFWatcher into a comprehensive, out-of-the-box Intrusion Detection System capable of handling larger volumes of target files.

Summary updated

BibTeX
@article{zhan2017protecting,
  author = {Dongyang Zhan and Lin Ye and Binxing Fang and Xiaojiang Du and Zhikai Xu},
  title = {{Protecting Critical Files Using Target-Based Virtual Machine Introspection Approach}},
  journal = {IEICE Trans. Information and Systems},
  year = {2017},
  doi = {10.1587/transinf.2016inp0009},
}
Figure