Instability in Downstream Task Performance During LLM Pretraining

Yuto Nishida, Masaru Isonuma, Yusuke Oda

Abstract

When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score. However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model. In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora. We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels. To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance volatility. We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.

The Problem: The Checkpoint Lottery

While the training loss of Large Language Models (LLMs) typically follows a smooth convergence curve, performance on actual downstream tasks—such as mathematical reasoning or question answering—often exhibits erratic fluctuations. This volatility creates what can be described as a "checkpoint lottery": evaluating a model at a local peak may lead to overestimating its capabilities, while a local valley might result in the premature dismissal of a promising training run.

Downstream task performance fluctuations across various benchmarks during pretraining.

This instability undermines the reliability and reproducibility of LLM research. The authors analyzed models ranging from 150M to 13B parameters to understand the root causes of these fluctuations and whether scaling model size provides a natural remedy.

Methods: Post-hoc Checkpoint Integration

To quantify instability, the researchers introduced two metrics: Mean Total Variation (MTV), measuring average score changes between consecutive checkpoints, and the Instability Score (IS), a reference-free metric tracking raw output changes.

To mitigate this instability, the study explores two post-hoc integration methods that do not require retraining:

  1. Checkpoint Averaging: Creating a single "smoothed" model by mathematically averaging the weights ($\theta$) of the $n$ most recent checkpoints.
  2. Checkpoint Ensemble: Deciding the final answer via a majority vote across multiple recent checkpoints.
Conceptual comparison between vanilla checkpoint evaluation and the proposed integrated checkpoint approach.

Theoretical analysis using first-order Taylor expansions suggests that weight averaging effectively approximates an average of task scores, thereby reducing variance.

Experimental Setup

Models were pretrained on the LLM-jp Corpus v3, a 2.1-trillion-token dataset of Japanese, English, and source code. Evaluation was conducted using the llm-jp-eval suite, covering nine diverse categories including Natural Language Inference (NLI), Mathematical Reasoning (MR), and Reading Comprehension (RC).

Key Insights

  • Scaling is Not a Cure-all: Increasing model size to 13B parameters does not eliminate performance fluctuations. Instability appears to be a fundamental characteristic of the pretraining process across all tested scales.
  • The "Flicker" Effect: At the example level, models frequently flip-flop between correct and incorrect answers across training steps. Aggregate instability is the result of these accumulated individual "flickers."
  • Weight Averaging as a "Free Lunch": Checkpoint averaging not only stabilizes scores but also provides a slight boost to mean performance. Because it results in a single model, it adds no inference-time overhead.
  • Metric Sensitivity: Discrete metrics (like Exact Match) show higher instability than continuous ones (like COMET for Machine Translation), suggesting that evaluation methodology plays a role in perceived volatility.

Results and Stability Gains

For the 13B parameter model, increasing the averaging window significantly reduced score variance. In the "Human Examination" (HE) category, the standard deviation decreased from 0.021 to 0.015. Overall performance also improved, with the aggregate score rising from 0.477 to 0.484 when using a window of 20 checkpoints. Qualitative analysis confirmed that integration methods successfully "iron out" erratic predictions, leading to smoother learning trajectories.

Limitations and Future Work

This study focused on base pretrained models. Future research is needed to determine if these findings extend to models after instruction tuning or RLHF. Additionally, while simple averaging was effective, more sophisticated "confidence-weighted" averaging techniques could potentially yield even greater stability and performance gains.

Summary updated

Abstract

When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score. However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model. In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora. We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels. To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance volatility. We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.
BibTeX
@inproceedings{nishida2025instability,
  author = {Yuto Nishida and Masaru Isonuma and Yusuke Oda},
  title = {{Instability in Downstream Task Performance During LLM Pretraining}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=xCu0LpuNlz},
}
Figure