Abstract
A whitelisting approach is a promising solution to prevent unwanted processes (e.g., malware) getting executed. However, previous solutions suffer from limitations in that: 1) Most methods place the whitelist information in the kernel space, which could be tempered by attackers; 2) Most methods cannot prevent the execution of kernel processes. In this paper, we present VAW, a novel application whitelisting system by using the virtualization technology. Our system is able to block the execution of unauthorized user and kernel processes. Compared with the previous solutions, our approach can achieve stronger security guarantees. The experiments show that VAW can deny the execution of unwanted processes effectively with a little performance overhead.
Introduction
Traditional application whitelisting—the practice of only allowing pre-approved programs to run—is a powerful defense against malware. However, existing implementations have two major vulnerabilities. First, most tools store their "allow list" and enforcement logic within the OS kernel. If an attacker exploits a kernel vulnerability, they can modify the whitelist in memory to authorize malicious code. Second, many tools focus only on user-level applications and fail to block unauthorized kernel processes or malicious kernel modules.
VAW (Virtualization-based Application Whitelisting) addresses these issues by hooking directly into the operating system's scheduler and leveraging hardware-assisted virtualization to isolate the whitelist and scheduling logic from the rest of the kernel.
System Architecture and Methods
VAW is implemented on the Xen hypervisor and relies on three main pillars to ensure tamper-proof enforcement:
- Scheduler-Based Enforcement: Instead of checking a file only when it is first opened, VAW modifies the Linux kernel's Completely Fair Scheduler (CFS). Every time the OS attempts to switch to a new process, the modified scheduler verifies if that process is on the whitelist. If unauthorized, the scheduler skips it in favor of an authorized process.
- The SIM Framework: To prevent attackers from tampering with the scheduler or the whitelist, VAW uses the Secure-In-VM (SIM) framework. This creates a separate, protected "scheduler address space" isolated from the rest of the guest kernel using Shadow Page Tables (SPT).
- Transition Gates: To maintain performance, VAW uses "transition gates"—special code regions executable in both spaces—and hardware features like
CR3_TARGET_LISTto switch address spaces rapidly without trapping into the hypervisor for every switch.
Processes are added to the whitelist via a trusted communication channel (hypercalls) using a secret key. To prevent hijacking during kernel thread creation, VAW uses watchpoints to monitor the execution path of internal kernel functions like copy_process(), ensuring the kernel followed a legitimate execution path.
Evaluation and Key Results
The system was evaluated for both security effectiveness and performance overhead. In security tests, VAW successfully blocked unauthorized user processes spawned via shellcode and prevented unauthorized kernel threads from being registered or executed.
Performance was measured using micro-benchmarks (LMbench) and macro-benchmarks (Linux kernel compilation). While the system introduces some latency, the overhead remains manageable for high-assurance environments.
As shown in the table above, context switching saw an overhead of approximately 2.39 µs. In macro-benchmarks, real-world performance hits included a 15% overhead for kernel decompression and a 14.5% overhead for a full kernel build. While significant for general-purpose computing, this is often considered an acceptable trade-off for the security guarantees provided.
Discussion and Limitations
VAW provides continuous enforcement at the "heartbeat" of the OS. By placing checks inside the scheduler, a process must be validated every time it requests CPU time. The "In-VM" isolation approach provides a middle ground between the high security of out-of-VM monitoring and the high performance of native execution.
However, the system currently assumes a "clean slate" state at boot; if the system is compromised during the boot process, the whitelist could be corrupted. Additionally, the requirement for kernel modifications and the ~15% performance overhead suggest that VAW is currently best suited for specialized, high-security server environments rather than general consumer use. Future work will focus on further optimizing address space transitions to reduce this overhead.
Summary updated