ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism

Tenghui Ma, Jihu Guo, Wei Gao, Sitian Lu, Zhisheng Ye, Hanjing Wang, Dahua Lin

TL;DR

ResiHP is a resilient training system designed to keep large-scale Large Language Model (LLM) training efficient despite inevitable hardware failures. By combining a workload-aware failure detector with a scheduler that dynamically adapts across multiple dimensions of parallelism, ResiHP improves training throughput by up to 4.39× compared to existing state-of-the-art systems.

What Problem Does This Paper Address?

Training modern LLMs requires massive clusters of thousands of GPUs, typically organized using hybrid parallelism—a mix of Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP). At this scale, hardware failures are a statistical certainty. These failures fall into two categories:

  1. Fail-stop: A GPU crashes or runs out of memory, immediately halting its work.
  2. Fail-slow: A GPU remains functional but performs significantly slower due to issues like thermal throttling or network jitter.

Fail-slow failures are particularly "insidious" because they don't trigger immediate errors but cause a cascading slowdown. Because hybrid parallelism relies on tight synchronization, one slow GPU can stall an entire pipeline and eventually the whole cluster.

Existing solutions struggle with two main issues:

  • Workload Noise: LLM datasets have varying sequence lengths. Even with "sequence packing," the quadratic cost of self-attention means some iterations naturally take longer than others. Existing systems often mistake these natural fluctuations for hardware failures, leading to "false alarms."
  • Coarse Adaptation: When a failure is detected, current systems often throw away perfectly healthy GPUs (e.g., discarding an entire TP group because one GPU failed) or create new imbalances while trying to fix old ones.

Methods

ResiHP introduces a decoupled architecture consisting of a Detector and a Scheduler.

Figure 4: ResiHP Overview.

1. Workload-Aware Detector

To avoid false positives, the Detector uses a lightweight analytical simulator. It models the expected execution time of each micro-batch by accounting for both linear computation (MLP layers) and the quadratic cost of self-attention based on actual sequence lengths. If an iteration is slow, the Detector checks if the slowness matches the predicted workload; if it doesn't, it flags a hardware failure.

2. Progressive Scheduler

Once a failure is confirmed, the Scheduler adapts the training setup across three levels:

  • TP-Level: Instead of discarding a whole group, it selectively excludes only the failed/slow devices and reconfigures the TP degree (e.g., from TP=4 to TP=2) to keep healthy GPUs working.
  • PP-Level: It performs Layer Repartitioning, assigning fewer model layers to the stages containing slow GPUs to balance the pipeline's rhythm.
  • DP-Level: It uses Progress-Aware Workload Migration, dynamically moving micro-batches from slow DP replicas to faster ones based on real-time execution progress.

Datasets

The authors evaluated ResiHP using two major open-source LLM families:

  • LLaMA 2: Variants with 7B, 13B, 30B, and 70B parameters.
  • Qwen 2.5: Variants with 7B, 14B, 32B, and 72B parameters.

Experiments were conducted on a cluster of 256 NVIDIA A100 GPUs. Failures were realistically simulated by manually terminating workers (fail-stop), locking GPU frequencies (fail-slow compute), and injecting network contention (fail-slow communication).

Key Insights and Ideas

  • The Failure Amplification Effect: A localized failure is not just a local problem. The authors quantified that one degraded GPU in a 32-GPU job can delay 31 other healthy GPUs, amplifying idle time by over 25× at the global synchronization boundary.
Failure amplification: A single failure propagates through TP, PP, and DP hierarchies, affecting the entire cluster.
  • Attention is the Culprit for Noise: Even when total tokens per batch are equalized, the way they are packed matters. One 4K sequence is much "heavier" than four 1K sequences due to $O(N^2)$ attention costs. Modeling this is the key to accurate failure detection.
Figure 1: Illustration of the impact of sequence length vari- ability on iteration-time fluctuations.
  • Heterogeneous Parallelism is Viable: ResiHP proves that you don't need a perfectly uniform cluster. By optimizing peer-to-peer communication between groups with different TP degrees, the system can salvage "fragmented" resources that other systems would simply waste.

Key Results

  • High Accuracy: The Detector achieved 99.4% accuracy in identifying failures while significantly reducing false alarms compared to the previous state-of-the-art (Greyhound).
  • Throughput Gains: ResiHP improved training throughput by 1.04–4.39× over baselines like ReCycle and Oobleck under mixed failure scenarios.
  • Robustness: In extreme scenarios where failures occurred every 30 minutes, ResiHP was the only system able to sustain training, whereas baselines eventually aborted.
  • Low Overhead: The system's "planning" phase takes less than 2 seconds, and the continuous monitoring adds only about 1.2–1.5% overhead to iteration times.

Limitations and Future Work

  • Silent Data Corruptions (SDCs): The current system focuses on liveness and timing signatures. It does not yet detect "silent" errors where a GPU produces incorrect mathematical results without slowing down. The authors suggest integrating loss-spike or parameter-drift detectors in the future.
  • Profiling Dependency: The execution time predictor requires an initial warm-up phase to profile hardware-specific costs, which might need repeating if the underlying hardware environment changes drastically.
  • Framework Specifics: While the concepts are portable, the current implementation is optimized for Megatron-LM-like frameworks. Applying this to other distributed backends might require significant engineering effort for communicator reconstruction.

Summary updated

Abstract

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual devices lead to performance skew across devices, diminishing overall training efficiency. Existing resilient systems overlook sequence length variability in datasets and device performance skew under hybrid parallelism. As a result, (1) iteration time fluctuations induced by sequence length variability can trigger spurious fail-slow detections, and (2) failures are mitigated through individual adaptations in hybrid parallelism, leading to unnecessary detection overhead and inefficient resilient training. To respond, this paper presents ResiHP, a resilient system that enables robust failure detection and fine-grained adaptation for hybrid parallel training. First, we develop a Detector to accurately identify failures. In particular, it employs a workload-aware execution time predictor that disentangles failures from iteration time fluctuations while remaining lightweight for online detection. Second, we design a Scheduler that dynamically adapts parallelism group sizes, model partitioning, and workload scheduling policies to improve training efficiency under failures. Experiments show that ResiHP improves training throughput by 1.04-4.39$\times$ compared with state-of-the-art resilient training systems under diverse failure scenarios in a 256-GPU cluster.
BibTeX
@inproceedings{ma2026resihp,
  author = {Tenghui Ma and Jihu Guo and Wei Gao and Sitian Lu and Zhisheng Ye and Hanjing Wang and Dahua Lin},
  title = {{ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism}},
  booktitle = {HPDC},
  year = {2026},
  doi = {10.1145/3806645.3807815},
}
Figure