TL;DR
This paper introduces FaRMv2, a distributed transactional system that brings "opacity" (ensuring transactions behave predictably even if they abort) to high-performance data centers. By leveraging precisely synchronized global time and efficient RDMA networking, FaRMv2 achieves near FaRMv1's performance while significantly simplifying distributed application development.
What Problem Does This Paper Address?
Developing distributed applications is notoriously complex. Developers often dream of a single, reliable machine that runs transactions perfectly, never failing and always behaving predictably. This ideal is called "opacity," which guarantees that transactions (even aborted ones) see a consistent view of the data and that their execution order aligns with real time. Existing systems either sacrifice this strong guarantee for performance or are too slow for modern data centers. FaRMv1 provided high performance and strict serializability for committed transactions but lacked opacity, forcing developers to write complex defensive code. FaRMv2 aims to bridge this gap by adding opacity to FaRMv1's high-performance foundation.
Methods
FaRMv2 builds upon FaRMv1's architecture, which uses optimistic concurrency control and one-sided RDMA for efficient data access. The key innovations for achieving opacity are:
- Global Time Synchronization: FaRMv2 synchronizes clocks across machines to within tens of microseconds using RDMA. This is significantly more precise than systems like Spanner and avoids the need for external hardware like GPS. It uses a clock master election and failover protocol to maintain global monotonicity even if the master fails.
- Timestamp Ordering with Uncertainty Handling: Transactions are serialized based on real-time timestamps. The system accounts for clock synchronization uncertainty by introducing "uncertainty waits" before acquiring read and write timestamps. This ensures that transactions see a consistent snapshot and that write locks are held until the write timestamp is definitively in the past.
- Multi-versioning: FaRMv2 supports multi-version concurrency control (MVCC) to reduce aborts in read-only transactions caused by conflicting writes. Old versions of objects are stored in memory and garbage collected.
- Fast-forward Failover: A protocol to quickly recover global time monotonicity after a clock master failure without unbounded uncertainty growth.
Datasets
- TPC-C: A standard benchmark for transactional database systems, used to evaluate throughput and scalability.
- YCSB: A benchmark for evaluating key-value stores, used to test performance under varying skew and conflict rates.
No new benchmarks were introduced in this paper.
Key Insights and Ideas
- Opacity is Achievable with High Performance: The paper demonstrates that providing strong opacity guarantees doesn't necessarily mean sacrificing data center performance. FaRMv2 achieves this by carefully integrating global time synchronization and uncertainty handling with efficient networking.
- Global Time is Crucial for True Opacity: Unlike systems that rely on local clocks or less precise synchronization, FaRMv2's tightly synchronized global time is fundamental to its ability to provide opacity and strict serializability.
- Developer Experience Matters: The authors highlight that the complexity introduced by the lack of opacity in FaRMv1 led to significant developer burden and code complexity. FaRMv2's opacity directly addresses this, simplifying the development of distributed applications.
- RDMA is Key to Low Latency Synchronization: The paper emphasizes how RDMA's low latency and CPU efficiency are critical for achieving the microsecond-level clock synchronization required for FaRMv2's performance goals.
Key Results
- High Throughput: FaRMv2 achieves 5.4 million neworder transactions per second on the TPC-C benchmark using 90 machines with 3-way replication. This is presented as the highest throughput for a system offering opacity and high availability.
- Low Opacity Overhead: The performance overhead of opacity in FaRMv2 is modest: a 3.6% reduction in throughput and a 69 µs increase in 99th percentile latency for short transactions compared to a non-opaque baseline.
- Scalability: Throughput scales well with cluster size, reaching 5.4 million neworders/s on 90 machines. Clock synchronization uncertainty increases only moderately with scale.
- High Availability: FaRMv2 recovers to full throughput within tens of milliseconds of a server failure.
- Multi-versioning Benefits: For workloads with significant read-only transactions and updates, MVCC in FaRMv2 can improve performance by reducing aborts, though it incurs a small cost in allocation and copying.
Limitations and Future Work
- Uncertainty Wait: While FaRMv2 significantly reduces clock synchronization uncertainty, it still requires "uncertainty waits" that add latency, especially for short transactions. The paper discusses a counterexample showing why skipping this wait breaks serializability.
- Early Aborts: FaRMv2 allows for "early aborts" in certain failure scenarios (e.g., primary failure) where old versions might not be immediately available. While this reduces common-case overhead, it requires applications to handle aborts gracefully.
- External Time Synchronization: FaRMv2's "fast-forward" mechanism for clock recovery can cause divergence from external time. While this is acceptable for many data center applications, the paper notes that "smearing" could be used if strict adherence to external time is required.
- Future Work: The paper suggests that further work could explore more aggressive re-replication for faster recovery at the cost of more contention, and investigate the impact of different clock synchronization sampling ratios on uncertainty. The authors also mention that the Al graph database, built on FaRMv2, demonstrates its practical utility.
Summary updated