MATCH: An MPI Fault Tolerance Benchmark Suite

Luanzheng Guo, Giorgis Georgakoudis, Konstantinos Parasyris, Ignacio Laguna, Dong Li

Introduction

As high-performance computing (HPC) systems transition toward exascale, system complexity increases while reliability decreases. Next-generation supercomputers are expected to face significantly higher failure rates; for example, the Sequoia system has reported a mean time between node failures of less than 20 hours. Most Message Passing Interface (MPI) applications are "fail-stop," meaning a single process failure causes the entire execution to crash, necessitating a full restart from the last checkpoint.

While various fault tolerance techniques—such as checkpointing and recovery frameworks like User-level Fault Mitigation (ULFM) and Reinit—have been proposed, the community lacked a structured method for comparison. This paper introduces MATCH, the first comprehensive benchmark suite designed to characterize and evaluate MPI fault tolerance designs across diverse scientific workloads.

The MATCH Framework

MATCH provides a unified environment to test combinations of checkpointing and recovery strategies. It integrates three core components:

  1. Checkpointing via FTI: The suite utilizes the Fault Tolerance Interface (FTI), a multi-level checkpointing library. FTI allows applications to save state to local storage, neighboring nodes, or parallel file systems, optimizing for different failure scenarios.
  2. Recovery Frameworks: MATCH implements and compares three primary recovery designs:
  3. RESTART-FTI: The traditional baseline where the application terminates and restarts completely upon failure.
  4. ULFM-FTI: Utilizes User-level Fault Mitigation to repair MPI communicators "online," avoiding a full process restart.
  5. REINIT-FTI: Employs the Reinit framework, which handles MPI state recovery transparently at the runtime level.
  6. Automated Data Identification: To reduce the developer burden of manual checkpointing, MATCH includes an analytic tool based on data dependency analysis. This tool automatically identifies the essential data objects required for a successful resume after a crash.

Experimental Setup

The researchers evaluated MATCH using six representative HPC proxy applications from the Exascale Computing Project (ECP) and LLNL suites: AMG (algebraic multi-grid), CoMD (molecular dynamics), LULESH (hydrodynamics), miniFE (finite element), miniVite (graph community detection), and HPCCG (conjugate gradient). These applications were tested across scaling sizes up to 512 processes on 32 nodes.

Comparative Performance Analysis

The evaluation revealed significant differences in efficiency and scalability between the recovery frameworks across the proxy applications.

Performance comparison of RESTART-FTI, REINIT-FTI, and ULFM-FTI across six HPC proxy applications.

The "Heartbeat" Tax and Transparency

A key finding was the overhead introduced by ULFM. ULFM implements a constant heartbeat mechanism for failure detection, which adds latency to normal communication even in failure-free runs. Furthermore, ULFM requires extensive application-level code modifications (often over 200 lines). In contrast, REINIT-FTI is implemented at the MPI runtime level, requiring fewer than five lines of code changes and incurring zero overhead during normal execution.

Scalability and Speed

Reinit demonstrated superior scalability, with recovery times remaining largely independent of the number of processes or problem size. Conversely, ULFM recovery times tended to increase as the system scaled. On average, Reinit recovery was found to be 4 times faster than ULFM and 16 times faster than a full Restart. In extreme cases, a full restart was up to 22 times slower than the Reinit method. FTI’s checkpointing also proved efficient, with L1 (local) checkpointing accounting for only about 13% of total execution time.

Conclusion

The MATCH benchmark suite provides critical insights into the future of HPC reliability. By demonstrating that the combination of FTI checkpointing and the Reinit recovery framework offers a high-performance, low-overhead, and scalable solution, MATCH establishes a new "gold standard" for evaluating fault-tolerant MPI designs. Future work will focus on automating FTI code generation and exploring local forward recovery strategies to further enhance resilience in exascale environments.

Summary updated

BibTeX
@inproceedings{guo2020match,
  author = {Luanzheng Guo and Giorgis Georgakoudis and Konstantinos Parasyris and Ignacio Laguna and Dong Li},
  title = {{MATCH: An MPI Fault Tolerance Benchmark Suite}},
  booktitle = {IISWC},
  year = {2020},
  url = {https://openreview.net/forum?id=fnlytClEWN},
}
Figure