TL;DR
While tracking downstream task performance during LLM pretraining is standard practice for checkpoint selection, evaluation scores frequently fluctuate wildly from step to step—even late in training. This paper demonstrates that scaling up model parameters does not eliminate this volatility, and shows both theoretically and empirically that simple, post-hoc checkpoint integration (averaging or ensembling neighboring checkpoints) drastically stabilizes evaluation and slightly improves overall accuracy without requiring changes to training.
What Problem Does This Paper Address?
During LLM pretraining, engineers and researchers track validation loss and downstream task benchmarks to select the "best" checkpoint. While training loss curves generally look smooth, downstream task metrics (e.g., accuracy on multiple-choice QA, reading comprehension, or reasoning) exhibit noisy, short-term fluctuations. A checkpoint saved at step $t$ might score significantly higher than at $t+1000$, only to drop at $t+2000$.
This erratic behavior creates major issues:
- Unreliable model selection: Picking a "peak" checkpoint may simply capture lucky noise rather than genuine capability gains.
- Poor reproducibility: Benchmarking becomes sensitive to the exact step saved.
- Unclear scaling dynamics: It was previously unclear whether larger models naturally grow out of this instability.
The authors set out to systematically quantify this downstream instability across training steps, test whether model scaling fixes it, and explore post-processing techniques to smooth out the noise.
Methods
- Quantifying Instability:
- Mean Total Variation (MTV): Measures the average step-to-step absolute change in reference-based task scores across consecutive checkpoints: $$\text{MTV}(\Theta) = \frac{1}{m-1} \sum_{t=1}^{m-1} |L(\theta_{t+1}) - L(\theta_t)|$$
-
Instability Score (IS): A complementary, reference-free metric assessing output-level volatility by computing the dissimilarity between consecutive generated outputs: $$\text{IS}(\Theta) = \frac{1}{m-1} \sum_{i=1}^{m-1} (1 - \text{sim}(x_{\theta_i}, x_{\theta_{i+1}}))$$ Both metrics were computed over the final 20% of training steps.
-
Post-Hoc Checkpoint Integration:
- Checkpoint Averaging: Computes a running uniform average of model weights across the most recent $n$ checkpoints ($\bar{\theta}_t = \frac{1}{n} \sum_{i=t-n+1}^t \theta_i$). This incurs zero additional inference cost.
- Checkpoint Ensembling: Gathers predictions from the individual $n$ checkpoints and applies majority voting (applicable to multiple-choice formats).
- Theoretical Justification: Using a first-order Taylor expansion around adjacent parameters (valid late in training where step-to-step parameter delta is small), the authors prove via the triangle inequality that checkpoint averaging mathematically guarantees reduced or equal variation: $\text{MTV}(\bar{\Theta}) \le \text{MTV}(\Theta)$.
Datasets
The authors analyzed autoregressive Transformer models pretrained on LLM-jp Corpus v3 (~2.1 trillion tokens combining Japanese, English, and source code).
Evaluation utilized the llm-jp-eval v1.4.1 benchmark suite across nine task categories in a 4-shot setup:
- MC (Multiple Choice QA): JCommonsenseQA, JCommonsenseMorality, KUCI
- HE (Human Examination): MMLU, JMMLU
- NLI: Jamp, JaNLI, JNLI, JSeM, JSICK
- QA & RC: JEMHopQA, NIILC, JSQuAD
- Other categories: Entity Linking (chABSA), Fundamental Analysis (Wikipedia Annotated Corpus), Mathematical Reasoning (MAWPS), and Machine Translation (ALT, WikiCorpus).
No new datasets were introduced.
Key Insights and Ideas
- Model scaling does not cure downstream instability: Evaluating models from 150M up to 13B parameters revealed that MTV and IS remain high regardless of model size. Larger models are just as prone to short-term score bouncing as smaller ones.
- Example-level flip-flopping: Instability is not merely an aggregate artifact. Inspecting individual examples revealed persistent alternations between correct and incorrect predictions across consecutive checkpoints late into pretraining.
- Averaging acts as a regularizer without inference penalties: Checkpoint weight averaging over a sliding window ($n \in \{2, 3, 5, 10, 20\}$) consistently suppressed MTV and IS while improving average scores.
Key Results
- Downstream Variance Reduction (13B Model): Over the final 20% of pretraining, raw vanilla checkpoints achieved an overall score of $0.477 \pm 0.006$. Applying checkpoint averaging with a window size of 20 improved the overall mean score to $0.484 \pm 0.003$ (halving the variance).
- Category-Level Gains:
- Entity Linking (EL): Increased from $0.403 \pm 0.025$ (vanilla) to $0.424 \pm 0.013$ (averaging, $n=20$).
- Multiple Choice (MC): Increased from $0.507 \pm 0.029$ to $0.521 \pm 0.020$.
- Question Answering (QA): Increased from $0.579 \pm 0.012$ to $0.595 \pm 0.009$.
- Averaging vs. Ensemble: While majority-vote ensembling effectively stabilized example-level predictions, weight averaging delivered larger improvements in mean accuracy across tasks while preserving single-model inference speed.
Limitations and Future Work
- Scope of Models and Corpora: Experiments were conducted on a single multilingual model family (LLM-jp) pretrained on a single corpus distribution.
- Pretraining Only: The study does not investigate whether downstream instability persists after supervised fine-tuning (SFT) or RLHF/alignment.
- Naive Weighting: The integration methods use uniform weighting across checkpoints. Future research could explore adaptive or uncertainty/confidence-weighted averaging schemes.
Summary updated