TL;DR
This paper introduces "speculative recovery," a novel fault-tolerance technique for disaggregated storage that significantly improves application availability compared to existing methods. By parallelizing failure detection and recovery, it drastically reduces downtime without sacrificing the cost-efficiency of disaggregated storage.
What Problem Does This Paper Address?
In modern cloud computing, disaggregated storage (where storage is separate from compute) is becoming ubiquitous. This enables a cost-effective fault-tolerance approach called Recovery from Disaggregated Storage (REDS). With REDS, if a primary application instance fails, a backup instance recovers its state from the disaggregated disk. While cheaper than traditional application-level replication (which requires multiple instances running simultaneously), REDS suffers from slow failover times. This is because REDS must first detect the primary's failure and then initiate recovery on the backup, leading to potentially long periods of unavailability, especially if the primary's failure is transient. The paper aims to bridge this gap by providing the high availability of replication with the cost-efficiency of REDS.
Methods
The core innovation is speculative recovery, which tackles the sequential nature of REDS. Instead of waiting for definitive failure detection, speculative recovery speculatively starts the recovery process on a backup instance as soon as the primary appears unavailable. This is achieved through two new disaggregated storage primitives:
super: This primitive allows for the creation of a temporary, isolated "superposition" of a disk. It enables a parent disk (on the primary) and a child disk (for the backup) to diverge. Thesuperprimitive uses a novel "collocated-clone" mechanism, which reuses the parent's allocation table to keep child blocks on the same storage shards as their parents. This significantly improves the performance of copy-on-write (COW) operations, making them nearly as fast as normal writes, unlike traditional COW clones.collapse: This primitive ensures correctness by determining which disk (parent or child) should be externally visible. It uses a global "dirty bit" on the parent disk. If the parent disk has been written to sincesuperwas invoked, it means the primary is likely still alive or has made progress, so the speculative recovery on the backup is aborted. If the parent disk has not been written to, the backup is promoted as the new primary, and the old primary is deallocated.
This parallel approach means that either the primary recovers itself, or the backup completes recovery, and whichever happens first wins, minimizing downtime.
Datasets
The paper does not explicitly mention specific datasets in the traditional machine learning sense. Instead, it uses:
- Real-world workloads: The evaluation uses representative database applications like MySQL (with InnoDB), PostgreSQL, and MariaDB (with RocksDB).
- TPC-C workload: This standard benchmark is used with the
oltpbenchclient to simulate application traffic and measure performance. - Recovery traces: Pre-captured block-level traces of recovery operations (using
fio) are replayed to ensure consistent and comparable testing across different disk types.
Key Insights and Ideas
- Parallelism is key to availability: The fundamental insight is that by speculatively pursuing both the "primary is alive" and "primary has failed" paths in parallel, we can dramatically reduce the worst-case downtime.
- Storage primitives can enable new fault-tolerance paradigms: The
superandcollapseprimitives are not just optimizations; they are foundational to enabling speculative recovery. Their design, particularlycollocated-clone, is crucial for achieving performance close to normal disks. - Cost-efficiency doesn't have to mean low availability: Speculative recovery demonstrates that the cost benefits of REDS can be combined with significantly higher availability, making it a more attractive option for many applications.
- False positives are manageable: While speculative recovery can be triggered by transient issues (false positives), the system is designed to gracefully abort and fall back to REDS. The analysis shows that the overhead from these aborted recoveries is minimal in practice.
Key Results
- Disk-level performance: The
collocated-cloneimplementation ofsuperachieves disk-level write performance close to regular disks, significantly outperforming traditional COW clones (e.g., 7.1x slower for Ceph's native clone under high concurrency vs.superbeing only slightly slower). - Application recovery latency: Recovery on
superdisks is only slightly slower (around 13% on average) than on regular disks, making speculative recovery viable. - End-to-end failover latency: Speculative recovery consistently provides lower failover latency than REDS, especially when REDS uses longer timeouts. In scenarios with false positives (primary recovers), speculative recovery significantly reduces unavailability by allowing the primary to continue.
- Resource overhead: The average resource overhead for speculative recovery is estimated to be very low (around 0.004%), primarily due to the infrequent nature of server inaccessibility events.
Limitations and Future Work
- Failure model: The current design relies on writes to the parent disk to detect failures. This might not detect "fail-slow" or certain partial failures, potentially leading to repeated aborted recoveries. The authors suggest falling back to REDS with a timeout in such cases.
- Application correctness: Speculative recovery assumes crash-consistent applications. It does not inherently provide linearizability for applications that are not crash-consistent.
- Future Work: The paper suggests exploring speculative recovery for other cloud storage interfaces (e.g., network file systems, object stores) and investigating its applicability beyond crash-consistent applications. Further research into handling more complex failure modes and optimizing the
collapseprimitive could also be beneficial.
Summary updated