TL;DR
This paper proposes a "semantic-aware" remote replication system that intelligently switches between efficient asynchronous replication and robust synchronous replication based on application signals. This approach aims to overcome the performance penalties of traditional synchronous replication over long distances, which are increasingly mandated for disaster recovery.
What Problem Does This Paper Address?
Businesses are increasingly required to maintain secondary disaster recovery sites far from their primary locations (hundreds of miles) for regulatory and resilience reasons. Traditional synchronous replication, which guarantees zero data loss by ensuring data is written to both primary and secondary sites before acknowledging the write, becomes prohibitively slow due to network latency over these long distances. This performance hit can cripple applications. Asynchronous replication is faster but risks data loss. The challenge is to achieve both the safety of synchronous replication and the efficiency of asynchronous replication, especially for applications that have varying data consistency needs.
Methods
The core of the proposed solution is a "semantic-aware" remote replication system. Instead of treating all data writes equally, this system monitors application and file system behavior. It leverages the fact that modern file systems and applications already have mechanisms to signal the importance of certain writes (e.g., using fsync or opening files in synchronous mode).
The system's strategy is:
- Default to Asynchronous Replication: For most operations, the system uses asynchronous replication, which is less sensitive to network latency and allows for better network utilization.
- Switch to Synchronous on Demand: When the application or file system signals a need for stronger consistency (e.g., through
fsync,fdatasync, or opening a file withO_SYNC), the replication system switches to synchronous semantics for those specific writes. This ensures that critical data is replicated synchronously, while less critical data benefits from asynchronous efficiency.
The paper also discusses how file systems handle writes (synchronous vs. asynchronous modes, journaling) and how databases interact with file systems, highlighting that applications already manage data persistence intelligently.
Datasets
Not discussed in the paper. The authors mention experiments conducted on a commercial replication system and a Linux file system, but specific datasets are not detailed.
Key Insights and Ideas
- Network Latency is the Bottleneck: The paper clearly demonstrates that synchronous replication performance degrades significantly with increasing network latency, making it impractical for geographically dispersed disaster recovery sites.
- Applications Already Signal Importance: Modern file systems and applications provide explicit signals (
fsync,O_SYNC) to indicate when data must be written to disk persistently. These signals are the key to enabling semantic-aware replication. - Asynchronous Replication is Efficient: Asynchronous replication, by batching writes and not waiting for remote acknowledgments, can effectively utilize network bandwidth and is much less impacted by latency.
- Balancing Safety and Performance: The proposed system offers a pragmatic solution by combining the strengths of both synchronous and asynchronous replication, allowing applications to dictate the level of consistency required on a per-write or per-file basis.
Key Results
The paper presents experimental results showing:
- Synchronous Replication Performance: Even with modest delays (e.g., 5-10 ms), synchronous replication throughput drops significantly, falling far below the available 1 Gbps link capacity.
- Asynchronous Replication Performance: Asynchronous replication, even with delays equivalent to continental US distances (up to 80 ms), maintains throughput close to the link capacity, demonstrating its suitability for long-distance replication.
- File System Behavior: Experiments with Linux's ext3 file system and Postgresql show how synchronous and asynchronous writes, as well as
fsynccalls, impact kernel event sequences and disk writes. Databases, for instance, usefsyncto ensure ACID properties even when using asynchronous file system writes.
Limitations and Future Work
The authors acknowledge several points:
- Implementation Complexity: While the high-level design is presented, a detailed system design and implementation are still works in progress.
- File-to-Block Mapping: Implementing
fsyncandfdatasyncrequires the replication system to be aware of block-to-file mappings or be file-aware, which adds complexity. - Write Ordering: The system needs to handle potential inconsistencies arising from out-of-order writes, especially when different applications access the same file with different consistency modes. Write coalescing at the volume manager level is suggested.
- Network Awareness: Further optimizations could involve making the system more network-aware to better utilize capacity and potentially inform the file system for opportunistic flushing.
Future work will focus on evaluating the system's benefits for various real-world workloads and completing the detailed system design and implementation.
Summary updated