Wave: A Split OS Architecture for Application Engines

Jack Tigar Humphries, Neel Natu, Kostis Kaffes, Stanko Novakovic, Paul Turner, Hank Levy, David E. Culler, Christos KozyrakisarXiv 2024 · arXiv · OpenReview

Abstract

The end of Moore's Law is driving cloud providers to offload virtualization and the network data plane to SmartNICs to improve compute efficiency. Even though individual OS control plane tasks consume up to 5% of cycles across the fleet, they remain on the host CPU because they are tightly intertwined with OS mechanisms. Moreover, offloading puts the slow PCIe interconnect in the critical path of OS decisions. We propose Wave, a new split OS architecture that separates OS control plane policies from mechanisms and offloads the control plane policies onto a SmartNIC. Wave introduces a new host-SmartNIC communication API, state synchronization mechanism, and communication techniques that overcome the PCIe bottleneck, even for μs-scale workloads. Wave frees up host compute for applications and unlocks new optimization opportunities, including machine learning-driven policies, scheduling on the network I/O path, and reducing on-host interference. We demonstrate that Wave enables OS control planes that are competitive with on-host performance for the most difficult μs-scale workloads, saving up to 16 host cores for memory management and 8 cores for network RPCs.

Introduction

In modern cloud environments, a significant portion of CPU cycles is consumed by the "datacenter tax"—the overhead of the operating system managing resources like networking, memory, and thread scheduling. While specialized hardware has successfully offloaded the data plane, the control plane logic remains on the host CPU due to the high latency of the PCIe bus. Communicating across PCIe typically takes microseconds, which is prohibitive for OS decisions that must occur in nanoseconds.

Wave addresses this challenge by offloading userspace system software—such as thread schedulers and memory managers—to the ARM cores of modern SmartNICs. By separating policy from mechanism, Wave reclaims valuable host CPU cycles for customer workloads.

The Wave Architecture

Wave employs a split-OS architecture where "agents" run on the SmartNIC to handle high-level logic, while the host kernel retains the low-level mechanisms required to enforce those decisions. This separation allows the system to leverage the dedicated compute power of the SmartNIC without requiring a complete rewrite of the host operating system.

The Wave split-OS architecture separates control plane policies on the SmartNIC from enforcement mechanisms on the host.

As shown in the architecture diagram, Wave uses a hybrid communication model. It utilizes Memory-Mapped I/O (MMIO) for low-latency signaling and Direct Memory Access (DMA) for high-throughput data transfers. To optimize cache performance, Wave uses specific Page Table Entry (PTE) types, such as Write-Combining to batch host-to-NIC messages and Write-Through to allow the host to cache NIC-made decisions.

Overcoming the PCIe Bottleneck

The primary innovation in Wave is its ability to hide PCIe latency through "prestaging" and "prefetching." Instead of the host waiting to request a decision, the SmartNIC agent proactively predicts future needs and stages decisions in a shared memory buffer.

Wave's communication flow: the SmartNIC agent 'prestages' decisions to hide PCIe latency.

When a host core needs to perform a task, such as a thread context switch, the decision is already available in the local cache, effectively bypassing the microsecond-scale round-trip time of the PCIe bus. This "eager" approach allows the SmartNIC to stay one step ahead of the host's execution.

Key Results and Evaluation

The researchers evaluated Wave using high-performance workloads like RocksDB and synthetic benchmarks on an AMD Zen3 host paired with an Intel Mount Evans IPU. Key findings include:

  • Resource Recovery: Offloading the memory manager (SOL) saved 16 host cores, while offloading the RPC stack (Stubby) saved 8 host cores.
  • Performance Parity: Wave's offloaded scheduler performed within 1.1% to 7.4% of the native on-host version, even for demanding μs-scale tasks.
  • Hardware Efficiency: By offloading the scheduler, Wave eliminated the need for frequent "timer ticks" on the host. This allowed idle cores to enter deeper sleep states, enabling active cores to maintain higher "Turbo Boost" frequencies and improving VM compute performance by 11.2%.
  • Memory Optimization: The offloaded ML-based memory manager reduced RocksDB’s memory footprint by 79% with minimal performance impact.

Conclusion

Wave demonstrates that offloading the OS control plane to SmartNICs is not only feasible but highly beneficial for cloud efficiency. By hiding PCIe latency through predictive prestaging, Wave recovers significant host resources while maintaining performance competitive with local execution. As interconnect technologies like CXL evolve to provide better cache coherency, the potential for split-OS architectures to further optimize datacenter performance will only grow.

Summary updated

Abstract

The end of Moore's Law is driving cloud providers to offload virtualization and the network data plane to SmartNICs to improve compute efficiency. Even though individual OS control plane tasks consume up to 5% of cycles across the fleet, they remain on the host CPU because they are tightly intertwined with OS mechanisms. Moreover, offloading puts the slow PCIe interconnect in the critical path of OS decisions. We propose Tide, a new split OS architecture that separates OS control plane policies from mechanisms and offloads the control plane policies onto a SmartNIC. Tide has a new host-SmartNIC communication API, state synchronization mechanism, and communication mechanisms that overcome the PCIe bottleneck, even for $\mu$s-scale workloads. Tide frees up host compute for applications and unlocks new optimization opportunities, including machine learning-driven policies, scheduling on the network I/O path, and reducing on-host interference. We demonstrate that Tide enables OS control planes that are competitive with on-host performance for the most difficult $\mu$s-scale workloads. Tide outperforms on-host control planes for memory management (saving 16 host cores), Stubby network RPCs (saving 8 cores), and GCE virtual machine management (11.2% performance improvement).
BibTeX
@misc{humphries2024wave,
  author = {Jack Tigar Humphries and Neel Natu and Kostis Kaffes and Stanko Novakovic and Paul Turner and Hank Levy and David E. Culler and Christos Kozyrakis},
  title = {{Wave: A Split OS Architecture for Application Engines}},
  year = {2024},
  eprint = {2408.17351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.17351},
}
Figure