Plumbline: When the Loss Lies, Validating LLM Training Correctly

Ruochen Wen, Hongyang Wang, Siwei Li, Wenbo Chen, Xian Li, Jingguo Liu, Leo Fan, Yiming Yan, Qian Qu, Zheshen Wang, Benoit DumoulinarXiv 2026 · arXiv

TL;DR

Plumbline is a correctness validation system for Large Language Model (LLM) training that detects both "loud" crashes and "silent" failures where the loss curve looks normal but the model is broken. By modularly checking every stage of the training lifecycle—from data pipelines to specific layer kernels—against trusted references, it enables rapid onboarding of new models and reduces root-cause diagnosis time from days to minutes.

What Problem Does This Paper Address?

Training LLMs at scale is a high-stakes engineering challenge where a single fault can waste thousands of GPU hours. The industry currently relies heavily on the "loss curve" as a health indicator, but the authors argue that the loss often lies.

Errors can be "silent": a bug in a chat template, a transposed weight matrix during checkpoint conversion, or a precision mismatch in a specific attention kernel might still result in a falling loss curve, but produce a model that underperforms or fails to stop generating text. Furthermore, LLM training stacks are complex, involving deep dependencies between frameworks (like Megatron-LM), libraries, and hardware. When a run does fail, finding the root cause is notoriously slow, often requiring manual log analysis and expensive trial-and-error reruns.

Methods

Plumbline introduces a systematic methodology based on two principles: correctness first (validating against trusted references) and modularity (testing parts in isolation).

The system operates at three levels:

  1. Lifecycle Correctness: Checks the environment (image versions), data pipeline (tokenization/masking), checkpointing (save/resume bitwise matches), and serving (export vs. original checkpoint).
  2. Architecture Correctness (The Step Check): This is a key innovation. It compares a single training step's gradients and loss from the framework under test against a trusted reference implementation (e.g., official Hugging Face code) running in FP32 on a CPU.
  3. Convergence Correctness: Runs the full training recipe on small, representative workloads to ensure the model trains stably to the end.
Figure 1. Plumbline’s checks along one training run, grouped by where a fault sits, and its diagnosis tools. Each check is labeled with what it compares, the training it needs and where it runs. The bottom row explains the markers.

A unique aspect of Plumbline is its agentic operation. It uses LLM agents to plan the validation scans, propose hypotheses for failures, and even suggest fixes, while a human reviewer provides final approval.

Datasets

Plumbline uses several representative workloads to exercise different code paths:

  • GSM8K: Used for initial recipe stability checks.
  • Tülu 3: For general chat capabilities.
  • Dolci-Think-SFT: To test "thinking-mode" reasoning.
  • LongReward-10k: To exercise long-context handling and memory-intensive kernels.

The system was evaluated across 3 frameworks (AutoModel, Megatron-SWIFT, Miles) and 5 state-of-the-art models, including DeepSeek-V4 and Qwen3.8-Flash.

Key Insights and Ideas

  • The "Silent Failure" Epidemic: Over half (32 out of 61) of the problems found were silent. These models would have finished training and likely reached production with undetected defects if not for Plumbline.
  • Step Checks are Scalable: You don't need to run a full training job to find architecture bugs. By comparing a single step against a CPU reference, Plumbline can catch kernel bugs or parallelism errors in minutes.
  • Systematic Noise Bounds: Because floating-point math is non-deterministic, Plumbline doesn't look for exact matches in gradients. Instead, it calculates a dynamic "noise floor" based on precision and run-to-run variance, failing a check only when the error exceeds 4x that floor.
Figure 2. The step check. The same rows train one step in the framework’s training command with one setting changed and in the reference implementation (here the model’s Hugging Face code) on CPU. A layer group fails when s > 4 max(F1, F2), so each failure points to one layer group, configuration and length range. Dot positions are illustrative.

Key Results

  • Bug Detection: Found 61 unique problems across the tested framework-model combinations.
  • Upstream Impact: 42 of these problems were fixed or reported upstream to open-source maintainers, with 8 being first-time discoveries by the authors.
  • Diagnosis Efficiency: In one case study, an out-of-memory error on an 8-node cluster was diagnosed and traced to a specific superlinear operation in a sparse attention kernel using just a single-node check and a CPU run, taking only "half a node-hour."
  • Failure Patterns: The authors categorized the 61 problems into seven recurring patterns, such as "Checkpoint conversion alters the checkpoint" and "Layers break under parallelism."

Limitations and Future Work

  • Reference Dependency: The "Step Check" requires a trusted reference implementation. If a new model is released without a standard implementation (like a Hugging Face model card), Plumbline must fall back to less rigorous checks.
  • Human-in-the-loop: While agents handle much of the heavy lifting, a human reviewer is still required to confirm problems and approve plans. The authors aim for full automation in future versions.
  • Scope: The step check only looks at the first few layers to save time; while this catches most layer-type bugs, it might miss faults that only emerge in deeper layers or after many steps (though convergence checks are meant to mitigate this).

Summary updated

Abstract

Correctness is a prerequisite for training large language models (LLMs) at scale, yet it is hard to ensure: a fault can sit in any part of a run, and how it shows differs across frameworks and the libraries they depend on. Some failures are loud and some are silent, and both take time to detect and trace. A single fault can waste a long run: the run crashes late, or the model scores lower only at the final evaluation. We present Plumbline, a correctness validation system for LLM training, used in production at Amazon. Plumbline checks each part of the training lifecycle, from the data a run trains on to the model it serves, down to the new layers each model brings and up to whole runs. Its diagnosis reuses these checks to trace failures to their root causes. Both rest on a systematic correctness methodology designed to carry across models and frameworks. Plumbline supports two agentic use cases: Day-0 onboarding, which validates the training of a newly released model before teams rely on it, and root-cause tracing of training failures. Across 3 frameworks and 5 models, it found 61 problems, 32 of them silent. Of these, 42 are fixed or reported upstream, 8 of them first raised by us. We further group the 61 into seven failure patterns that show where training breaks, and report how problems recur across frameworks and models.
BibTeX
@misc{wen2026plumbline,
  author = {Ruochen Wen and Hongyang Wang and Siwei Li and Wenbo Chen and Xian Li and Jingguo Liu and Leo Fan and Yiming Yan and Qian Qu and Zheshen Wang and Benoit Dumoulin},
  title = {{Plumbline: When the Loss Lies, Validating LLM Training Correctly}},
  year = {2026},
  eprint = {2611.13393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13393},
}
Figure