TL;DR
As LLM training scales to tens of thousands of GPUs, hardware defects like silicon aging cause "Silent Data Corruption" (SDC) that can quietly ruin model quality without crashing the system. This paper introduces LLM-PRISM, the first framework to map physical, register-level GPU defects to software-level training behavior across 7,664 full pre-training runs. The authors reveal that while LLMs naturally absorb low-frequency faults, specific numerical formats (like FP16) and fault locations can trigger catastrophic, invisible model degradation that standard safety checks completely miss.
What Problem Does This Paper Address?
State-of-the-art LLMs require massive clusters running continuously for weeks or months. At this scale, hardware reliability becomes a statistical certainty: Google and Meta have reported that SDC events occur every week or two during major training runs.
Unlike "fail-stop" errors that cause a system crash, SDCs are insidious. They stem from manufacturing escapes or silicon wear-out, manifesting as permanent or intermittent hardware faults that subtly corrupt computations. Prior research heavily focused on transient faults (one-off bit flips from cosmic rays). This paper tackles permanent and intermittent faults, which systematically recur over time. Until now, we lacked a hardware-grounded understanding of how these persistent physical defects propagate through the complex optimization landscape of LLM pre-training.
Methods
The authors developed LLM-PRISM (Permanent and Intermittent SDC Modeling), a framework that bridges the gap between low-level hardware design and high-level software training loop dynamics. The methodology consists of two primary steps:
- RTL-Level Fault Characterization: Rather than guessing where bit-flips happen, the authors ran cycle-accurate simulations on the actual Register-Transfer Level (RTL) design of an NVIDIA H100 GPU. They replayed real training traces through key functional units (Tensor Cores, CUDA cores, caches, and register files) and injected physical "stuck-at" defects to extract over 6,000 realistic hardware error signatures.
- Software-Level Fault Injection: They embedded a stochastic injection engine directly into NVIDIA’s production-grade Megatron-LM framework.
To accurately replicate physical defects in software, they parameterized the fault site into a 7-element tuple spanning spatial, temporal, and numerical dimensions (including Rank, Layer, Training Phase, and Error Rate). Intermittent faults were modeled using a Bernoulli trial at each training iteration.
Datasets
The models (GPT-2 Small 124M and Medium 355M) were pre-trained from scratch using the WikiText language modeling dataset. To evaluate whether training-time corruptions translated to real-world capability loss, the final models were tested on downstream benchmarks via the LM Evaluation Harness, specifically the Children’s Book Test (CBT) and the Winograd Schema Challenge.
Key Insights and Ideas
- Four Distinct Failure Modes: The paper expands our understanding of hardware faults beyond the simple "recover vs. crash" binary. They identify four regimes:
- Spike-and-recover: Low-frequency faults cause temporary loss spikes that the optimizer absorbs.
- Spike-and-degrade: High-frequency faults collapse the loss scale, stalling the optimizer.
- Silent degradation: The loss curve looks stable but settles at a permanently worse baseline.
- Gradual drift: The loss curve is visually indistinguishable from a clean run, but the model's internal weights diverge drastically over time.
- Numerical Formats Dictate Vulnerability: The choice of precision format (FP16, BF16, FP8) fundamentally changes how a hardware fault manifests.
- FP8 acts as a natural fault container; its narrow dynamic range forces faults to either saturate harmlessly or trigger an immediate NaN crash.
- FP16 is highly vulnerable to silent degradation because its wide dynamic range allows large, corrupted values to propagate without triggering overflow detectors.
- BF16 represents an awkward middle ground, combining high crash susceptibility with substantial silent degradation.
- Not All Training Phases Are Equal: Faults affecting backward weight gradients (
bwd_grad_weights) are largely self-correcting due to optimizer momentum and gradient averaging across multiple GPUs. Conversely, faults in forward outputs (fwd_out) and backward input gradients (bwd_grad_inp) act locally before cross-rank communication, propagating completely unattenuated and causing a ~60% crash rate.
Key Results
The authors executed 7,664 complete training runs (totaling roughly 11,500 GPU hours).
- High Baseline Resilience: Across all formats, 57% to 76% of runs completed with a final perplexity (PPL) indistinguishable from the fault-free baseline, proving LLM training is robust against low-rate intermittent faults.
- The Danger of the Tail: At a fault rate of 1.0 (permanent stuck-at), the median PPL for FP16 remained near baseline (28.58), but the 95th percentile exploded to 192.4, demonstrating severe tail risk.
- The Blind Spot of NaN Detectors: Standard runtime monitoring (skipping steps when a
NaNorInfis detected in the loss) successfully prevents outright crashes in FP16 and BF16. However, it leaves extreme silent degradation outliers completely untouched. - Downstream Capability Loss: Models categorized under "Gradual Drift" (with normal-looking loss curves) still suffered measurable accuracy drops on downstream tasks like CBT and Winograd, proving that standard loss-curve monitoring is insufficient for ensuring model quality.
Limitations and Future Work
The primary limitation is scale: due to the computational cost of running thousands of full training trajectories, the evaluation was restricted to GPT-2 scale models (124M/355M parameters). While smaller models allowed for rigorous statistical profiling, massive foundation models might exhibit different outlier dynamics or quantization sensitivities.
Additionally, the study focused on a specific distributed layout using sequence parallelism. Future work is needed to explore how alternative distributed training strategies (like pipeline or tensor parallelism) amplify or attenuate the "blast radius" of an SDC event across a cluster.
Summary updated