TL;DR
This paper introduces NEWMADELEINE, a communication library designed for high-performance computing that tackles network failures. It achieves this through a novel combination of quick network failure detection and low-overhead checksums for message integrity, significantly improving resilience without substantial performance degradation.
What Problem Does This Paper Address?
Modern supercomputers are increasingly composed of hundreds of thousands of interconnected components. This massive scale dramatically increases the probability of hardware or software failures, including issues within the network itself (e.g., switch failures, NIC crashes). While checkpointing can handle node failures, the communication layer needs to be robust to network disruptions to maintain connectivity and ensure message integrity. Existing solutions often incur significant overhead, impacting application performance. This paper aims to provide a fault-tolerant communication library that is both generic and high-performance, addressing the critical need for resilience in large-scale computing environments.
Methods
The NEWMADELEINE library implements two core fault-tolerance mechanisms:
- Generic Retransmission Mechanism: This mechanism reroutes communication flow through surviving network links when a failure is detected. It leverages the fact that many supercomputers have multiple network interfaces (e.g., high-speed networks and Ethernet), allowing for graceful fallback.
- Quick Network Failure Detection: Instead of relying on slow, multi-second timeouts, NEWMADELEINE uses a timer-interrupt-driven approach, guaranteeing detection within a few scheduler timeslices (typically 10ms). This rapid detection is crucial for minimizing memory consumption associated with message logging.
- Low-Overhead Checksums: To ensure message integrity, NEWMADELEINE employs checksums. The key innovation here is amortizing the computational cost by interleaving checksum calculation with data copying operations, especially during the "eager" send protocol. For larger messages using the "rendezvous" protocol, checksums are computed in parallel using multiple threads.
Datasets
Not discussed in the paper. The evaluation uses standard benchmarks.
Key Insights and Ideas
- Network failures are a critical bottleneck: While node failures are often addressed, network issues can be equally disruptive and require dedicated solutions within the communication layer.
- Speed of detection matters: Rapid network failure detection is paramount to reducing the memory footprint of message logging mechanisms, which are essential for retransmission.
- Checksums can be integrated efficiently: By interleaving checksum computation with data movement (copying), the overhead can be significantly reduced, making end-to-end integrity checks more practical.
- Leveraging multiple network paths is key: The ability to reroute traffic across available links is a fundamental strategy for surviving single network failures.
Key Results
- Low Overhead: The fault-tolerant NEWMADELEINE shows minimal performance degradation compared to its non-fault-tolerant version. For example, on Infiniband, the message retransmission mechanism adds only 500 ns overhead, and message integrity checks add an extra 100 ns.
- Competitive Performance: In evaluations, NEWMADELEINE FT+CSum (fault-tolerant with checksums) often performs comparably to or better than OPEN MPI's fault-tolerant components, especially in throughput tests.
- NAS Parallel Benchmarks: While some kernels (BT, LU, SP) show a slight performance decrease due to the sender waiting for acknowledgments in the rendezvous protocol, overall, the impact on most benchmarks is manageable. OPEN MPI's fault-tolerance mechanisms, in contrast, showed a more significant overhead on several kernels.
Limitations and Future Work
- Limitations:
- The paper acknowledges that in some "rendezvous" communication patterns, the sender waiting for acknowledgments can introduce performance delays.
- While the paper focuses on network failures, it implicitly assumes that the underlying drivers can provide some level of error reporting.
- Future Work:
- The authors plan to investigate integrating these fault-tolerance techniques into higher layers of the software stack, such as parallel file systems (e.g., PVFS), which also rely on reliable communication.
- Further study into the integration of these techniques within upper layers of the software stack is planned.
Summary updated