In traditional operating system security, the kernel represents the ultimate boundary of trust. Applications run in an unprivileged user space, relying on the kernel to enforce isolation. However, this design suffers from a fundamental flaw: if an attacker compromises the kernel through an exploit, the entire system's security architecture collapses.
To break this all-or-nothing security paradigm, researchers have turned to hardware virtualization. By leveraging hardware extensions originally designed to run multiple operating systems on a single physical machine (such as Intel VT-x or ARM Virtualization Extensions), security architects can establish a higher privilege level—often conceptualized as "Ring -1" or EL2—that operates entirely underneath or outside the guest operating system.
This architectural shift works because the virtualization layer controls the hardware primitives directly. It manages physical memory mappings via hardware-enforced page tables, traps sensitive CPU instructions, and isolates execution contexts. As a result, even if a guest operating system is completely compromised, it remains trapped within its virtualized boundaries, unable to tamper with security mechanisms running at a higher privilege level.
Establishing the Virtualization Boundary
The foundational intuition behind using virtualization as a security boundary was formalized by Tal Garfinkel et al. in their seminal work on Terra: a virtual machine-based platform for trusted computing. The authors introduced the concept of a Trusted Virtual Machine Monitor (TVMM), a high-assurance software layer that partitions a single hardware platform into multiple isolated virtual machines (VMs). By separating workloads into "open-box" environments (for general-purpose applications) and "closed-box" environments (for secure, tamper-resistant applications), they demonstrated that security semantics could be enforced at the hardware boundary, completely isolated from the platform owner or adjacent workloads.
As hardware support for virtualization matured and expanded to mobile and embedded architectures, these boundaries became significantly more lightweight. For instance, Dong Shen et al. designed TinyVisor: An extensible secure framework on android platforms by leveraging ARM’s hardware virtualization extensions. By establishing a hypervisor that runs at a higher privilege level than the Android guest OS, the framework can transparently intercept system events and monitor guest behavior without relying on the integrity of the mobile kernel.
Out-of-VM Monitoring and Intrusion Tolerance
Moving the security boundary to the hypervisor layer enables a powerful design pattern: out-of-VM monitoring. Because the security logic resides outside the guest VM, it is completely invisible and inaccessible to attackers operating within the guest environment.
This external perspective is highly effective for detecting and mitigating attacks. For example, Chuanyi Liu et al. developed POSTER: TraceVirt: A Framework for Detecting the Non-tampering Attacks in the Virtual Machine, which utilizes hardware capabilities like Extended Page Tables (EPT) or Nested Page Tables (NPT) to control and monitor memory access from outside the target VM. Because the detection modules operate within the host virtualization layer, they can identify sophisticated exploits without modifying the guest or risking compromise from within.
Similarly, Alexander Pretschner demonstrated in A virtualized usage control bus system that placing a communication infrastructure for data usage control at the hypervisor level provides a tamper-resistant enforcement mechanism. Because the bus operates beneath the guest OS, users and processes within the guest cannot bypass or alter the security policies. Umar Syed reinforced this in Inmate OS and VMM based Secured Virtualization for Cloud Encompassment, showing that shifting security logic into the Virtual Machine Monitor (VMM) provides strict resource insulation, ensuring that host security instructions always take precedence over guest commands.
This isolation is equally vital for recovering from attacks. Shengzhi Zhang et al. leveraged VM boundaries in Availability-sensitive intrusion recovery to isolate intrusion recovery components from infected guest operating systems, allowing for safe damage assessment, state replay, and migration. At cloud scale, Xuefeng Zheng et al. showed in Sensitive Data Protection Based on Intrusion Tolerance in Cloud Computing how a privileged management domain within a hypervisor can transparently intercept network traffic and coordinate state updates across active-passive replicas, keeping the coordination logic entirely secure from compromises occurring within individual guest environments.
Inverting the Threat Model: Protecting Guests from the Host
While early virtualization security focused on protecting the host from the guest, the rise of multi-tenant public clouds inverted the threat model: how do you protect a user's virtual machine from a malicious cloud administrator or a compromised hypervisor?
Traditional hypervisors are large, complex software suites containing millions of lines of code for hardware emulation and lifecycle management. This massive attack surface means a compromise in the hypervisor can expose every tenant VM on the physical machine. To mitigate this, researchers began deprivileging and splitting the hypervisor layer.
Wuqiong Pan et al. proposed this architectural split in Improving Virtualization Security by Splitting Hypervisor into Smaller Components. By breaking a bloated hypervisor into smaller, per-VM management components, they removed complex emulation code from the global root of trust. If an attacker exploits a vulnerability in a specific VM's management layer, the blast radius is confined to that single instance, preventing lateral movement to other tenants.
Taking this deprivileging strategy to its logical conclusion, Jiahao Chen et al. introduced delegated virtualization in Security and Performance in the Delegated User-level Virtualization. Their architecture handles runtime VM exits directly within unprivileged user space rather than the host kernel, confining potential hypervisor exploits to a highly restricted environment with negligible performance overhead.
Traditional Virtualization: Nested Virtualization (CloudVisor):
+-----------------------------------+ +-----------------------------------+
| Guest VMs (Tenant Workloads) | | Guest VMs (Tenant Workloads) |
+-----------------------------------+ +-----------------------------------+
| Hypervisor (Massive TCB) | | Hypervisor (Untrusted Control) |
+-----------------------------------+ +-----------------------------------+
| Hardware | | Security Monitor (Tiny TCB) |
+-----------------------------------+ +-----------------------------------+
| Hardware |
+-----------------------------------+
When the primary hypervisor cannot be modified or trusted at all, nested virtualization can be used to insert a thin layer of trust beneath it. Fengzhe Zhang et al. demonstrated this approach with CloudVisor: retrofitting protection of virtual machines in multi-tenant cloud with nested virtualization. By introducing a tiny, high-assurance security monitor underneath the hypervisor, CloudVisor intercepts memory management and I/O operations, ensuring the privacy and integrity of tenant VM data even if the primary hypervisor is fully compromised. This principle of treating the infrastructure management layer as an untrusted adversary is also reflected in A Trusted-based Cloud Computing Virtual Storage System and Key Technologies by Kehe Wu et al., where security controllers are isolated within separate VM boundaries to protect sensitive cryptographic operations from cloud administrators.
Fine-Grained Intra-VM Isolation and Hardware Enclaves
Modern hardware has evolved to support fine-grained privilege separation inside a single virtual machine, giving rise to Confidential Virtual Machines (CVMs). Technologies like AMD SEV-SNP introduce hardware-enforced Virtual Machine Privilege Levels (VMPLs), allowing a single VM to be subdivided into distinct security domains.
In The Road to Trust: Building Enclaves within Confidential VMs, Dan Meng 0002 et al. demonstrated how VMPLs can act as a hardware primitive to build secure enclaves within a CVM. By running a security monitor and an enclave emulation layer at the highest privilege level (VMPL0) while the primary guest OS operates at a lower privilege level (VMPL1 or lower), the hardware prevents a compromised or untrusted guest OS from accessing the memory space of the secure enclaves.
This multi-level privilege architecture is also being applied to specialized accelerators. Chenxu Wang et al. developed Building a Lightweight Trusted Execution Environment for Arm GPUs, which leverages secure virtualization extensions to run security monitors at a higher privilege level than the primary secure operating system. This ensures that the integrity of GPU computations remains intact even if the primary secure OS is compromised.
However, strict isolation introduces a trade-off: it makes standard infrastructure maintenance tasks (such as live migration or snapshotting) exceptionally difficult, because the host control plane cannot inspect or modify the guest data plane. To bridge this gap without breaking the security boundary, Jiahao Chen et al. proposed CPC: Flexible, Secure, and Efficient CVM Maintenance with Confidential Procedure Calls. Confidential Procedure Calls allow host-initiated maintenance modules to execute securely within the guest environment, maintaining the strict cryptographic boundaries of confidential computing while restoring operational flexibility.
Alternative Abstractions and Conceptual Extensions
The success of hypervisor-enforced isolation has inspired alternative virtualization abstractions that re-think the relationship between the operating system and the hardware. For instance, Yiming Zhang et al. introduced the "process-like VM" (pVM) abstraction in KylinX: A Dynamic Library Operating System for Simplified and Efficient Cloud Virtualization. Instead of running a full guest OS inside a VM, KylinX treats the hypervisor as the core operating system and individual Unikernel appliances as lightweight processes. This design combines the strong, hardware-enforced isolation of virtualization with the performance and agility of traditional processes.
Finally, the architectural philosophy of hierarchical privilege separation has proven so powerful that it has been adapted metaphorically outside the domain of systems security. In ALIS: Aligned LLM Instruction Security Strategy for Unsafe Input Prompt, Xinhao Song et al. applied this concept to Large Language Models (LLMs). By structuring model prompts into a privilege hierarchy—treating system-level instructions as a high-priority "kernel" and user inputs as unprivileged "user space"—the authors demonstrated that the model could systematically veto adversarial jailbreaks and prompt injections. Whether implemented in silicon or software logic, the principle remains identical: true security requires a higher privilege layer capable of overriding and isolating untrusted execution.
Key Foundational Work
Terra: a virtual machine-based platform for trusted computing (2003) — contributed 2 concept elements to this direction.
Availability-sensitive intrusion recovery (2009) — contributed 1 concept elements to this direction.