TL;DR
This paper introduces MDev-NVMe, a novel NVMe storage virtualization solution that leverages mediated pass-through to achieve near-native performance within virtual machines. By combining full NVMe virtualization with an active polling mechanism, MDev-NVMe significantly boosts throughput and reduces latency compared to existing solutions, making NVMe storage more accessible and efficient in cloud environments.
What Problem Does This Paper Address?
High-performance computing and cloud environments increasingly rely on fast storage solutions like NVMe SSDs. However, traditional virtualization methods often introduce significant performance overhead, leading to NVMe devices being underutilized. Existing solutions like Virtio offer broad compatibility but suffer from performance limitations, while direct pass-through (VFIO) provides near-native speed but lacks device sharing capabilities, hindering scalability and VM migration. This leaves a gap for a virtualization solution that can deliver high NVMe performance, enable efficient device sharing among multiple VMs, and maintain low latency, all without requiring guest OS modifications.
Methods
MDev-NVMe employs a mediated pass-through approach, which is a full virtualization mechanism. Key aspects of its design include:
- Native NVMe Driver Support: VMs can use their existing, unmodified NVMe drivers, simplifying adoption and avoiding guest OS changes.
- Device Emulation: The MDev-NVMe kernel module emulates essential PCI device features, including registers and interrupts (INTx, MSI, MSI-X), allowing VMs to interact with a virtual NVMe device.
- Admin Queue Emulation: A single pair of Admin Submission Queues (SQ) and Completion Queues (CQ) is emulated for the host. This allows multiple VMs to manage their NVMe operations, with MDev-NVMe translating virtual admin commands into physical ones.
- I/O Queue Shadowing: Instead of emulating each I/O queue, MDev-NVMe shadows guest I/O queues to physical queues. This allows guest I/O commands to be executed directly by the device, significantly reducing overhead.
- DMA & LBA Translation: A mechanism is implemented to translate guest physical addresses (GPA) to host physical addresses (HPA) for DMA buffers and to adjust Logical Block Addresses (LBA). This is crucial for resource partitioning and security.
- Active Polling Mode: To overcome the overhead associated with interrupts and "vm-exits" during frequent I/O operations, MDev-NVMe introduces a 2-way polling mode. This involves polling for doorbell updates and completion entries, reducing the need for host kernel interrupts and improving performance.
Datasets
The paper does not introduce new datasets. Instead, it uses real-world NVMe SSDs for evaluation:
- Intel OPTANE SSD DC P4800X: A high-performance NVMe SSD with 3D XPoint Memory media.
- Intel SSD DC P3600: A more commonly used NVMe SSD.
The evaluation is performed using the Flexible I/O tester (FIO), a standard tool for I/O benchmarking. The paper defines specific FIO test cases (Table 1) covering various scenarios like random read/write with different queue depths and mixed read/write workloads, using direct I/O without a file system.
Key Insights and Ideas
- Mediated Pass-Through as a Sweet Spot: The paper argues that mediated pass-through offers a compelling balance between the performance of direct pass-through and the sharing capabilities of para-virtualization, addressing the limitations of both.
- Polling is Crucial for High-Performance NVMe Virtualization: The authors highlight that traditional interrupt-driven mechanisms are a bottleneck for NVMe's inherent parallelism. Active polling, despite its CPU utilization, is essential for achieving near-native performance in demanding I/O workloads.
- Guest OS Independence is Key: Allowing VMs to use their native NVMe drivers simplifies integration and broadens the applicability of the solution.
- Queue Shadowing Reduces Overhead: By avoiding full emulation of I/O queues and instead shadowing them, MDev-NVMe significantly cuts down on the virtualization overhead.
- Block Size Matters: The choice of I/O block size has a significant impact on performance, with a sweet spot around 4K-16K for balancing throughput and latency. Larger blocks can introduce overhead due to hypervisor-level separations.
Key Results
MDev-NVMe demonstrates significant performance improvements across various metrics:
- Throughput: Achieves up to 142% improvement over native interrupt-driven throughput and over 2.5x the throughput of Virtio. It shows the best throughput performance among the tested virtualization mechanisms.
- Latency: Exhibits 70% of native average latency and 31% of Virtio average latency. It also provides a bounded maximum latency and promising QoS performance, with competitive tail latency.
- Scalability: MDev-NVMe shows promising scalability, maintaining performance as the number of VMs increases, with queue assignment becoming the bottleneck at high VM counts.
- Block Size Influence: MDev-NVMe performs well with block sizes around 4K-16K, offering better throughput and latency than native or SPDK solutions in this range.
Limitations and Future Work
- CPU Utilization for Polling: The active polling mechanism, while boosting performance, can lead to higher CPU utilization on the host server, especially with many VMs.
- Queue Scheduling Overhead: The simple FIFO policy for scheduling I/O queues can introduce overhead in heavy I/O workloads, potentially leading to balancing issues.
- Future Work:
- Developing more sophisticated scheduling algorithms or a "queue resource ballooning" methodology to address scheduling overhead.
- Exploring adaptive polling modes to dynamically adjust polling aggressiveness based on workload demands.
- Designing future high-performance I/O device hardware to better support or cooperate with polling algorithms.
- Further optimization of the polling algorithm for MDev-NVMe.
Summary updated