Abstract
Large Language Model (LLM) training on massive GPU clusters is frequently interrupted by hardware failures, leading to significant productivity losses. Traditional metrics like uptime or failure rate fail to capture the hidden costs of recovery, such as rollback waste and checkpointing overhead. This paper introduces the Training Overhead Ratio (TOR), a comprehensive metric that quantifies the ratio of optimal training time to observed training time. By modeling the performance preservation ratio $r(t)$ across different failure scenarios (fail-stop and fail-slow), TOR provides a practical framework for engineers to estimate realistic training timelines and optimize system reliability.
Introduction
Training state-of-the-art LLMs is a months-long endeavor requiring thousands of GPUs. At this scale, hardware failures are inevitable. For instance, the training of Llama-3 encountered 466 interruptions over 54 days. Despite the frequency of these events, the industry lacks a standard metric to measure their impact on actual progress.
Existing metrics are often insufficient:
- Reliability/Failure Rate: Ignores the time required for recovery tasks.
- Availability: May count a system as "up" even if it is performing no useful work due to data corruption or slow recovery loops.
- Model FLOPs Utilization (MFU): Focuses on hardware efficiency during active computation but misses the "wasted" time between the last successful checkpoint and a system crash.
To bridge the gap between ideal and observed training times, we propose the Training Overhead Ratio (TOR).
Methodology
The core contribution is the mathematical definition of TOR: $$\text{TOR} = \frac{\text{Optimal Training Time}}{\text{Observed Training Time}}$$ TOR values range from 0 to 1, with 1 representing a failure-free system.
Performance Preservation Ratio $r(t)$
To calculate TOR, we introduce the performance preservation ratio $r(t)$, representing the proportion of optimal work achieved at time $t$. In a healthy state, $r(t) = 1$. During repair or rollback, $r(t) = 0$.
As shown in the figure above, the system transitions through various states:
- Healthy Run: Optimal performance ($r(t)=1$).
- Save Checkpoint: Periodic overhead to ensure progress is saved.
- Fail-stop: Immediate cessation of work, requiring a "Roll-Back" to the last checkpoint. The time between the last checkpoint and the failure is "Roll-Back Waste."
- Fail-slow: The system continues to run but at a degraded performance level ($R_{fs} < 1$) before eventually failing or being repaired.
Failure Modeling
By categorizing failures into fail-stop and fail-slow types, we derive closed-form equations for TOR based on the Mean Time Between Failures (MTBF). For a fail-stop system: $$\text{TOR}_{\text{fail-stop}} = \frac{\text{MTBF} - T_{sr}(1 - R_{sr}) - T_{rb} - N_{ckpt}T_{ckpt}}{\text{MTBF} + T_r}$$ where $T_{sr}$ is slow recovery time, $T_{rb}$ is rollback waste, $N_{ckpt}$ is the number of checkpoints, and $T_r$ is repair time.
Key Insights and Results
- Capturing the Rollback Penalty: TOR explicitly accounts for "garbage time"—the work lost between the last checkpoint and a crash. This is a critical factor in large-scale training that traditional availability metrics ignore.
- Addressing Fail-Slow Scenarios: Fail-slow nodes are often "silent killers" that degrade performance without triggering immediate alerts. TOR treats this reduced performance as a direct hit to reliability.
- Predictive Planning: TOR allows teams to move beyond best-case scenario planning. By knowing a cluster's MTBF and checkpoint overhead, engineers can calculate a realistic deadline by multiplying theoretical time by $1/\text{TOR}$.
Limitations and Future Work
While TOR provides a robust theoretical foundation, several areas remain for future exploration:
- Empirical Validation: Large-scale experimental validation across different hardware architectures (e.g., H100 vs. B200) is needed.
- Heterogeneous Failures: Future models could account for overlapping, non-uniform failure types through weighted TOR calculations.
- Checkpoint Optimization: While TOR can identify the impact of checkpoint frequency, specific algorithms are needed to find the "sweet spot" that balances checkpoint overhead against potential rollback waste.
Summary updated