Evaluating the Robustness of Chinchilla Compute-Optimal Scaling

Rylan Schaeffer, Noam Itzhak Levi, Andreas Kirsch, Theo Guenais, Brando Miranda, Elyas Obbad, Sanmi KoyejoSubmission to ICLR 2026 · Rejected

TL;DR

Recent scrutiny has raised doubts about Hoffmann et al.’s influential Chinchilla scaling laws, highlighting parameter ambiguities and wide confidence intervals. This paper reveals that even though Chinchilla’s reported model parameters diverge from standard architectural calculations by up to 15.2%, the core scaling results and the iconic “20 tokens per parameter” rule remain remarkably resilient. Through analytical proofs and stress tests across four perturbation models, the authors show that Chinchilla remains a dependable guide for allocating pretraining compute.

What Problem Does This Paper Address?

The Chinchilla scaling paper (Hoffmann et al., 2022) revolutionized large language model training by demonstrating that parameters and tokens should scale in roughly equal measure, giving rise to the influential "20-to-1" rule of thumb (20 training tokens per model parameter).

However, subsequent studies have questioned Chinchilla's reliability, citing wide confidence intervals, internal discrepancies among Chinchilla’s three analysis methods, and contradictions with Kaplan et al.’s earlier OpenAI scaling laws. These questions left practitioners uncertain: Can we still trust Chinchilla to guide multi-million-dollar training budgets?

This paper addresses this question in two steps:

  1. Identifying whether ambiguities in how Chinchilla counted model parameters alter its findings.
  2. Determining how large and what kinds of distortions in parameter counts would be required to break Chinchilla’s prescriptions.

Methods

The authors evaluate the parametric scaling law fitting the pretraining loss $L$ to parameters $N$ and dataset size $D$: $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

Their investigation proceeds along two main thrusts:

  1. Parameter Discrepancy Analysis: The authors compared the 50 model configurations from Chinchilla's Table A9 against standard transformer parameter formulas:

  2. Reported: Values printed in Table A9.

  3. Standard Formula: Calculated using standard attention ($4 \cdot d_\text{model} \cdot \text{kv\_size} \cdot n_\text{heads}$) and feed-forward formulas with tied embeddings.
  4. Best-Fit Formula: A formula replacing the factor of 4 with 5 in the attention projection term, which closely matches the reported numbers.
  5. Perturbation Stress Tests: Using the fitting codebase from Besiroglu et al. (2024) with 4,000 bootstrap resamples, the authors applied four structured perturbations to parameter counts:

  6. Multiplicative error: $\tilde{N} = c_m \cdot N$

  7. Additive error: $\tilde{N} = N + c_a$ (e.g., omitting or including embedding layers)
  8. Systematic bias: $\tilde{N} = \mu_{\text{geo}} (N / \mu_{\text{geo}})^s$ (scaling distortions between small and large models)
  9. Log-normal noise: $\tilde{N} = N \cdot \exp(\delta), \delta \sim \mathcal{N}(0, \sigma^2)$

Datasets

The paper does not train new models or introduce new natural language benchmarks. Instead, it utilizes:

  • The architectural hyperparameters and training evaluation data of the over 400 model pretraining runs (spanning 44M to 16B parameters) originally reported in Hoffmann et al. (2022).
  • The standardized parameter and loss extraction dataset and open-source fitting pipeline provided by Besiroglu et al. (2024).

Key Insights and Ideas

  • An Accidental "Factor of 5": The discrepancy between standard architectural parameter formulas and Chinchilla’s reported parameters exists for all 50 models in Table A9 (average error: 7.4%, maximum: 15.2%). Replacing 4 with 5 in the attention calculation accounts for 44 of the 50 discrepancies, suggesting a minor counting artifact in the original work.
  • The Scaling Laws Don't Notice: Refitting the power laws across all three parameter interpretations yields virtually identical fit parameters ($\hat{E}, \hat{A}, \hat{\alpha}, \hat{B}, \hat{\beta}$) well within bootstrap error bars. In fact, using the standard formula makes the optimal token-to-parameter ratio even flatter across compute scales than the original reported parameters.
  • Why Structure Dictates Sensitivity:
  • Multiplicative shifts preserve the compute-optimal exponent: they only rescale prefactor constants without altering the compute-invariance of the optimal token-to-parameter ratio.
  • Additive shifts and systematic biases alter the derivative in log-log space ($\frac{d \log g(N)}{d \log N} \propto \frac{N}{N + c_a}$ or $\hat{\alpha} = \alpha/s$). This changes whether larger compute budgets favor comparatively more or fewer tokens per parameter.

Key Results

  • Parameter Ambiguity: The relative discrepancy between Chinchilla’s reported parameters and standard formula calculations spans 3.6% to 15.2% (average 7.39%).
  • Token-to-Parameter Ratio Stability: The compute-optimal ratio remains centered at $\approx 20$ across all three parameter interpretations. The drift across training compute budgets is flattest using standard formula parameters (slope of $-0.572$ per decade of compute, versus $-1.049$ for best-fit and $-1.248$ for reported parameters).
  • Power-Law Decay under Bias: Under systematic slope bias $s$, the parameter exponent decays cleanly as a power law: $\hat{\alpha} = 10^{-0.46} \cdot s^{-1}$ ($R^2 > 0.999$, $p \approx 5.9 \times 10^{-90}$).
  • Noise Tolerance: The scaling fit tolerates substantial random log-normal noise; parameters only become unidentifiable at extreme noise levels ($\sigma \approx 3.162$).

Limitations and Future Work

  • Focus on Parametric Fitting: The study primarily re-examines Chinchilla’s parametric loss-fitting methodology (Approach 3) and does not rerun full pretraining sweeps.
  • Classic Compute Formulation: The analysis operates within the traditional pretraining compute regime ($C \approx 6ND$), without accounting for post-Chinchilla trade-offs.
  • Future Directions: The authors suggest applying similar robustness stress tests to modern paradigms, including inference-constrained scaling (e.g., Llama-style overtraining), data-constrained regimes with multi-epoch training, and test-time reasoning compute scaling.

Summary updated

Abstract

Hoffman et al. (2022)'s Chinchilla paper introduced the principle of compute-optimal scaling, laying a foundation for future scaling of language models. In the years since, however, valid concerns about Chinchilla have been raised: wide confidence intervals, discrepancies between its three approaches, and incongruities with other scaling laws. This raises a critical question for the field: Can practitioners still rely on Chinchilla's prescriptions? Our work demonstrates the answer is yes. We begin by uncovering that the model parameters central to Chinchilla's analyses were ambiguous: three interpretations are possible, with relative differences between different interpretations of model parameters as high as 15.2\%. We find that, perhaps surprisingly, which model parameters are used for the analyses do not meaningfully affect key results: the scaling law estimates and the compute-optimal tokens-to-parameter ratio. Indeed, under one interpretation, the tokens-to-parameter ratio becomes more constant with the target compute budget. We then ask how distorted the Chinchilla model parameters \textit{could} have been without meaningfully affecting the key results. By deliberately perturbing model parameters in four structured ways, we find that key Chinchilla results are most sensitive to additive or systematic errors, which can alter the otherwise flat trend of the optimal tokens-to-parameter ratio, but overall, Chinchilla's key results withstand sizable perturbations. Altogether, our findings offer the field renewed confidence in Chinchilla as a durable guide for scaling language models.
BibTeX
@misc{schaeffer2025evaluating,
  author = {Rylan Schaeffer and Noam Itzhak Levi and Andreas Kirsch and Theo Guenais and Brando Miranda and Elyas Obbad and Sanmi Koyejo},
  title = {{Evaluating the Robustness of Chinchilla Compute-Optimal Scaling}},
  year = {2025},
  eprint = {2509.23963},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23963},
}
Figure