TL;DR
MATCH is the first comprehensive benchmark suite designed to evaluate and compare different Message Passing Interface (MPI) fault tolerance designs. By testing various recovery strategies across six high-performance computing (HPC) proxy applications, the researchers discovered that combining the Fault Tolerance Interface (FTI) for checkpointing with the "Reinit" recovery framework provides a highly efficient, scalable solution that significantly outperforms traditional restart methods and the popular ULFM framework.
What Problem Does This Paper Address?
As supercomputers grow in size and complexity, their reliability decreases. Next-generation HPC systems are expected to face much higher failure rates; for instance, the Sequoia supercomputer reported a mean time between node failures of just 19.2 hours. Most MPI applications are "fail-stop," meaning if a single process or node fails, the entire execution crashes and must be restarted from scratch.
While several fault tolerance techniques exist—such as checkpointing (saving state to disk/memory) and various MPI recovery frameworks like User-level Fault Mitigation (ULFM) and Reinit—there has been no structured way to compare them. Developers lacked a "gold standard" benchmark to determine which combination of tools works best for specific scientific workloads. MATCH fills this gap by providing a unified framework to characterize and research these designs.
Methods
The authors developed MATCH, which integrates three primary components:
- Checkpointing: They used the Fault Tolerance Interface (FTI), a multi-level checkpointing library that can save data to local storage, neighboring nodes, or parallel file systems.
- MPI Recovery Frameworks: They implemented and compared three designs:
- RESTART-FTI: The baseline; the application stops and restarts completely after a failure.
- ULFM-FTI: Uses User-level Fault Mitigation to repair the MPI communicator "online" without a full restart.
- REINIT-FTI: Uses the Reinit framework, which transparently handles MPI state recovery at the runtime level.
- Automated Data Identification: To simplify the "programming tax" of checkpointing, the authors created an analytic tool based on data dependency analysis. This tool automatically identifies which data objects are essential to save to ensure the application can correctly resume after a crash.
Datasets (Workloads)
Instead of traditional datasets, this paper uses six representative HPC proxy applications from the Exascale Computing Project (ECP) and LLNL suites:
- AMG: An algebraic multi-grid solver.
- CoMD: A molecular dynamics proxy.
- LULESH: A hydrodynamics solver.
- miniFE: An unstructured implicit finite element proxy.
- miniVite: A graph community detection proxy.
- HPCCG: A conjugate gradient solver.
These apps were tested across four scaling sizes (up to 512 processes on 32 nodes) and three input sizes (small, medium, large).
Key Insights and Ideas
- The "Heartbeat" Tax: ULFM introduces background overhead even when no failure occurs. This is because it implements a constant heartbeat mechanism to detect failures and amends MPI interfaces, which adds latency to normal communication.
- Reinit’s Transparency: Unlike ULFM, which requires significant application-level code changes (often over 200 lines per benchmark), Reinit is implemented at the MPI runtime level. It requires fewer than five lines of code changes and incurs zero overhead during failure-free execution.
- Scalability Divergence: Reinit recovery time is independent of the number of processes and problem size. In contrast, ULFM recovery time increases as the system scales, making it less suitable for massive exascale systems.
- Checkpointing Efficiency: FTI’s L1 (local memory/SSD) checkpointing scales remarkably well, with the time spent writing checkpoints accounting for only about 13% of total execution time.
Key Results
- Recovery Speed: On average, Reinit recovery is 4 times faster than ULFM and 16 times faster than a full Restart.
- Worst Case Scenarios: In some tests, a full restart was up to 22 times slower than using Reinit.
- Failure-Free Performance: REINIT-FTI performed nearly identically to the baseline (RESTART-FTI) when no failures occurred, whereas ULFM-FTI showed visible performance degradation as the number of processes increased.
Limitations and Future Work
While the MATCH suite provides a powerful comparison tool, the authors acknowledge a few areas for growth:
- Code Generation: Their data dependency tool currently identifies where to checkpoint but does not yet automatically generate the necessary FTI code. This remains a manual step for developers.
- Recovery Strategies: The current evaluation focuses on global non-shrinking recovery. Future versions of MATCH could explore "local forward recovery," where only the failed processes are corrected while others continue working.
- Extensibility: While the methodology is extensible, the suite currently only includes six proxy apps. Adding more diverse workloads (like AI/ML training) would further validate these findings.
Summary updated