Tailwind: Fast and Atomic RDMA-based Replication

Yacine Taleb, Ryan Stutsman, Gabriel Antoniu, Toni CortesUSENIX ATC 2018 · OpenReview

TL;DR

This paper introduces Tailwind, a novel replication protocol that significantly boosts the performance of in-memory data stores by leveraging RDMA to eliminate CPU involvement and data copying on backup servers. By allowing backups to remain idle during replication, Tailwind frees up valuable CPU resources for primary servers, leading to substantial improvements in throughput and latency.

What Problem Does This Paper Address?

In-memory key-value stores are crucial for high-performance data-intensive applications, offering microsecond-level access times. However, ensuring fault tolerance through replication introduces significant overhead, consuming up to 80% of CPU cycles in write-intensive workloads. Existing replication methods, even those using RDMA, often involve redundant data copying and require active participation from backup servers, negating some of RDMA's efficiency benefits. Furthermore, achieving atomic data transfers with RDMA is challenging due to the potential for partially applied writes during failures, which can lead to data inconsistency. This paper tackles the problem of making replication in these high-performance systems efficient, atomic, and less resource-intensive.

Methods

Tailwind is a zero-copy, zero-CPU replication protocol designed for primary-backup systems. Its core innovation lies in its use of one-sided RDMA writes to transfer data directly from the primary server's DRAM to the backup server's I/O buffers, completely bypassing the backup server's CPU. This allows backup servers to remain passive and idle during the replication process.

To ensure atomicity and data consistency despite idle backups, Tailwind employs a novel protocol that detects incomplete RDMA transfers. This is achieved by appending checksums and metadata to the log data. During recovery, backups scan these logs to reconstruct metadata and verify data integrity.

The protocol separates the data path (using RDMA for high-throughput transfers) from the control path (using conventional RPCs for less frequent operations like buffer allocation, setup, and failure notifications). This hybrid approach simplifies complex operations without impacting the performance of data movement.

Datasets

Not discussed in the paper. The evaluation uses the YCSB benchmark with synthetic workloads.

Key Insights and Ideas

  • Idle Backups are Key: The most significant insight is that by making backup servers completely passive during replication (zero-CPU), substantial performance gains can be achieved. This frees up primary server resources and improves overall system throughput.
  • RDMA for Data, RPC for Control: A pragmatic approach of using RDMA for the heavy lifting (data transfer) and RPCs for less frequent, complex control operations (like buffer management and failure handling) offers a good balance of performance and simplicity.
  • Checksums for Atomicity: The clever use of checksums and metadata appended to the log data provides a robust mechanism for detecting incomplete RDMA transfers and ensuring data consistency, even when backups are idle. This is crucial for achieving strong consistency with the proposed method.
  • Scalability through Connection Management: Tailwind's connection management, which scales with the number of replicas rather than the entire cluster size, makes it a scalable solution for large distributed systems.

Key Results

  • Throughput Improvement: Tailwind improves RAMCloud's normal-case request processing throughput by up to 1.7x. For update-heavy workloads (50% PUTs), it achieves a 70% throughput improvement.
  • Latency Reduction: Tailwind cuts down write median latencies by 2x and 99th percentile latencies by 3x. Specifically, under light load, median update latency is reduced from 16 µs to 11.8 µs, and 99th percentile latency from 27 µs to 14 µs. Under heavy load, median latency drops from 32 µs to 16 µs, and tail latency from 78 µs to 28 µs.
  • Resource Efficiency: Tailwind significantly reduces CPU utilization for replication. For update-only workloads, it uses 3x fewer worker cores and 4x fewer dispatch cores compared to RAMCloud.
  • Crash Recovery Performance: Tailwind introduces no significant overhead during crash recovery compared to RAMCloud.

Limitations and Future Work

  • Metadata Overhead: While Tailwind adds only 4 bytes per object for metadata, this is still an overhead. However, the paper notes that log-cleaning mechanisms can efficiently manage this.
  • RDMA Dependency: The protocol inherently relies on RDMA-capable networks, limiting its applicability to systems with such infrastructure.
  • Future Work: The paper suggests that Tailwind can be integrated into other systems that use distributed logging and have RDMA support. The authors also mention that the protocol's design can be adapted for different consistency levels and durability guarantees. The current implementation focuses on primary-backup replication, and extending it to other replication topologies could be a future direction.

Summary updated

BibTeX
@inproceedings{taleb2018tailwind,
  author = {Yacine Taleb and Ryan Stutsman and Gabriel Antoniu and Toni Cortes},
  title = {{Tailwind: Fast and Atomic RDMA-based Replication}},
  booktitle = {USENIX ATC},
  year = {2018},
  url = {https://openreview.net/forum?id=NWAz9GC3ye},
}
Figure