TL;DR
Shared network datapaths in datacenters face a critical challenge: ensuring low tail latency for one tenant without sacrificing the efficiency of shared resources. This paper introduces Virtuoso, a novel time-protection mechanism that enforces per-tenant CPU-time budgets within these shared datapaths, achieving strong tail latency isolation with minimal overhead. Virtuoso's approach allows for efficient resource sharing while preventing "noisy neighbors" from impacting critical latency guarantees.
What Problem Does This Paper Address?
Modern datacenters rely heavily on shared software network datapaths for functions like virtual switching, network virtualization, and reliable transport. These datapaths are shared across multiple applications, containers, or virtual machines (VMs), often belonging to different tenants. While this sharing offers significant efficiency benefits by pooling resources and leveraging microarchitectural optimizations, it introduces a critical problem: tail latency isolation.
When multiple tenants share the same datapath cores, work done for one tenant can directly delay service for another. This is particularly problematic because packet processing in these datapaths is extremely fast (hundreds of CPU cycles) and operates on a "run-to-completion" model without preemption. Even small, infrequent delays caused by an adversarial tenant can significantly inflate the tail latency for a victim tenant, impacting performance-sensitive applications. Existing solutions either sacrifice efficiency by dedicating cores (coarse-grained partitioning) or offer weak isolation when sharing cores (fine-grained sharing with rate limits). This paper aims to bridge this gap by providing strong tail latency isolation within a shared datapath without compromising efficiency.
Methods
Virtuoso introduces a time-protection mechanism based on three core principles:
- Token Bucket with Time as Tokens: Instead of using packet or byte counts, Virtuoso uses CPU cycles as the "tokens" in a budget. Each tenant is assigned a CPU-time budget per datapath core. As packets are processed, the actual CPU time consumed is measured using the CPU timestamp counter (TSC) and deducted from the tenant's budget.
- Intervention Points and Bounded Deficit: Protection is enforced at specific "intervention points" within the datapath (e.g., packet reception (RX), transmission (TX), and scheduling (SCHED)). At these points, the system checks if a tenant has sufficient budget. If a tenant temporarily exceeds its budget, it accrues a "bounded deficit" that must be repaid later. This avoids preemption and allows for efficient batch processing.
- Delayed RX Attribution and Enforcement: For received packets, tenant attribution often happens after initial processing (e.g., after parsing headers). Virtuoso handles this by accounting for the CPU time spent on RX immediately but deferring strict enforcement until the next TX or SCHED intervention point. This prevents expensive packet re-queuing and maintains efficiency.
Virtuoso employs a split architecture: a fast path for latency-critical packet processing and a slower path for global coordination and budget replenishment.
Datasets
Not discussed in the paper. The evaluation uses a custom testbed with two identical machines configured as clients and servers, connected via 100 Gbps Ethernet adapters.
Key Insights and Ideas
- Tail Latency is a Time Problem, Not a Volume Problem: The core insight is that controlling tail latency requires managing CPU time directly, not just packet or byte throughput. Packet processing costs vary significantly due to factors like cache locality, branch prediction, and batch composition, making volume-based controls insufficient.
- Bounded Deficit for Efficiency and Protection: Allowing tenants to accrue small, bounded deficits instead of strictly enforcing budgets at every micro-operation is crucial. This preserves the efficiency of batching and run-to-completion execution while still bounding interference.
- Intervention Points are Key: Identifying and leveraging specific points in the datapath where work can be attributed to a tenant and enforcement can occur is fundamental to the design.
- Delayed RX Attribution is a Practical Challenge: The asymmetry between TX (where tenant is known) and RX (where it's often determined later) requires a specific mechanism to avoid penalizing tenants for work done before attribution.
Key Results
- Strong Tail Latency Isolation: In a case study with the TAS TCP stack, Virtuoso demonstrated a 7.8x reduction in victim tail latency under adversarial interference compared to unmodified shared TAS.
- High Efficiency: Virtuoso achieved per-core efficiency improvements of up to 3x compared to siloed datapaths under bursty workloads, while keeping throughput within 5% of unmodified shared TAS.
- Minimal Overhead: Virtuoso's changes to TAS resulted in only a slight increase in latency (median latency of 6µs vs. 7µs for TAS) and a small throughput reduction (5% for large messages).
- Scalability: Virtuoso scales well with the number of cores, showing performance close to shared TAS.
Limitations and Future Work
- Residual Microarchitectural Interference: Virtuoso does not claim to eliminate all residual microarchitectural interference (e.g., shared cache and TLB effects) that can persist across tasks.
- Receiver-Side Feedback Dependency: For open-loop traffic, receiver-side protection relies on mechanisms like post-attribution drops or explicit congestion notification (ECN) marking. Without these, the bound on unattributed work per RX batch is maintained, but zero cross-tenant perturbation isn't guaranteed if endpoints don't enforce similar budgets.
- Adaptive Control and Placement: The authors suggest tuning budget parameters dynamically rather than statically to better adapt to workload diversity. This could involve online profiling or application hints.
- Integration with Resource Management: Exploring how Virtuoso can be integrated with existing placement and load-balancing mechanisms is a promising direction.
- Hardware Support: Leveraging hardware features on DPUs and SmartNICs could further enhance Virtuoso's performance and isolation.
Summary updated