Abstract
The end of Moore's Law is driving cloud providers to offload virtualization and the network data plane to SmartNICs to improve compute efficiency. Even though individual OS control plane tasks consume up to 5% of cycles across the fleet, they remain on the host CPU because they are tightly intertwined with OS mechanisms. Moreover, offloading puts the slow PCIe interconnect in the critical path of OS decisions. We propose Wave, a new split OS architecture that separates OS control plane policies from mechanisms and offloads the control plane policies onto a SmartNIC. Wave introduces a new host-SmartNIC communication API, state synchronization mechanism, and communication techniques that overcome the PCIe bottleneck, even for μs-scale workloads. Wave frees up host compute for applications and unlocks new optimization opportunities, including machine learning-driven policies, scheduling on the network I/O path, and reducing on-host interference. We demonstrate that Wave enables OS control planes that are competitive with on-host performance for the most difficult μs-scale workloads, saving up to 16 host cores for memory management and 8 cores for network RPCs.
Introduction
In modern cloud environments, a significant portion of CPU cycles is consumed by the "datacenter tax"—the overhead of the operating system managing resources like networking, memory, and thread scheduling. While specialized hardware has successfully offloaded the data plane, the control plane logic remains on the host CPU due to the high latency of the PCIe bus. Communicating across PCIe typically takes microseconds, which is prohibitive for OS decisions that must occur in nanoseconds.
Wave addresses this challenge by offloading userspace system software—such as thread schedulers and memory managers—to the ARM cores of modern SmartNICs. By separating policy from mechanism, Wave reclaims valuable host CPU cycles for customer workloads.
The Wave Architecture
Wave employs a split-OS architecture where "agents" run on the SmartNIC to handle high-level logic, while the host kernel retains the low-level mechanisms required to enforce those decisions. This separation allows the system to leverage the dedicated compute power of the SmartNIC without requiring a complete rewrite of the host operating system.
As shown in the architecture diagram, Wave uses a hybrid communication model. It utilizes Memory-Mapped I/O (MMIO) for low-latency signaling and Direct Memory Access (DMA) for high-throughput data transfers. To optimize cache performance, Wave uses specific Page Table Entry (PTE) types, such as Write-Combining to batch host-to-NIC messages and Write-Through to allow the host to cache NIC-made decisions.
Overcoming the PCIe Bottleneck
The primary innovation in Wave is its ability to hide PCIe latency through "prestaging" and "prefetching." Instead of the host waiting to request a decision, the SmartNIC agent proactively predicts future needs and stages decisions in a shared memory buffer.
When a host core needs to perform a task, such as a thread context switch, the decision is already available in the local cache, effectively bypassing the microsecond-scale round-trip time of the PCIe bus. This "eager" approach allows the SmartNIC to stay one step ahead of the host's execution.
Key Results and Evaluation
The researchers evaluated Wave using high-performance workloads like RocksDB and synthetic benchmarks on an AMD Zen3 host paired with an Intel Mount Evans IPU. Key findings include:
- Resource Recovery: Offloading the memory manager (SOL) saved 16 host cores, while offloading the RPC stack (Stubby) saved 8 host cores.
- Performance Parity: Wave's offloaded scheduler performed within 1.1% to 7.4% of the native on-host version, even for demanding μs-scale tasks.
- Hardware Efficiency: By offloading the scheduler, Wave eliminated the need for frequent "timer ticks" on the host. This allowed idle cores to enter deeper sleep states, enabling active cores to maintain higher "Turbo Boost" frequencies and improving VM compute performance by 11.2%.
- Memory Optimization: The offloaded ML-based memory manager reduced RocksDB’s memory footprint by 79% with minimal performance impact.
Conclusion
Wave demonstrates that offloading the OS control plane to SmartNICs is not only feasible but highly beneficial for cloud efficiency. By hiding PCIe latency through predictive prestaging, Wave recovers significant host resources while maintaining performance competitive with local execution. As interconnect technologies like CXL evolve to provide better cache coherency, the potential for split-OS architectures to further optimize datacenter performance will only grow.
Summary updated