When training a deep learning model, the primary constraint is almost always the total computational budget, typically measured in floating-point operations (FLOPs). For a standard Transformer, this budget is roughly proportional to the product of the number of parameters ($N$) and the number of training tokens ($D$). If your computational budget doubles, how should you allocate it? Should you build a model that is twice as large, or train the same model on twice as much data?
For years, the prevailing heuristic leaned heavily toward scaling model size while keeping dataset sizes relatively modest. However, a foundational shift occurred when Jordan Hoffmann et al. demonstrated in An empirical analysis of compute-optimal large language model training that many prominent large language models were significantly undertrained. By mapping out the performance of over 400 models across varying budgets, the authors revealed that for a fixed training compute budget, parameters and tokens should scale in roughly equal proportions. A subsequent stress test by Rylan Schaeffer et al. in Evaluating the Robustness of Chinchilla Compute-Optimal Scaling confirmed that this balanced data-to-parameter ratio is remarkably robust to parameter counting ambiguities and experimental perturbations.
Why Equal Scaling Works: The Core Intuition
Why does this clean, proportional relationship emerge? Intuitively, a model's final error can be decomposed into two fundamental bottlenecks: representation capacity (limited by model size) and optimization or estimation error (limited by data volume). If you scale one dimension without the other, you hit diminishing returns. An enormous model trained on a tiny dataset will quickly memorize the noise rather than learning generalizable features, while a tiny model trained on a massive dataset will lack the capacity to absorb the underlying complexity of the data.
From an information-theoretic perspective, Benjamin Van Roy et al. proved in Information-Theoretic Foundations for Neural Scaling Laws that the optimal allocation under a fixed compute budget follows a linear relationship because it perfectly balances the rates of error decay from both sources. When these decay rates are balanced, the total error is minimized.
However, this clean symmetry can shift based on task complexity and optimization noise. Blake Bordelon used Dynamical Mean Field Theory in A Dynamical Model of Neural Scaling Laws to show that the benefits of increasing training duration versus model size are governed by data spectral decay. Similarly, Courtney Paquette et al. mapped this behavior in 4+3 Phases of Compute-Optimal Neural Scaling Laws, showing that while the square-root scaling of optimal model size with compute is a common emergent property, the exact ratio depends heavily on the interplay between representation capacity and optimization noise.
Architectural and Task Variations
The equal-scaling recipe is not a universal constant; it changes dramatically depending on the architecture and objective.
For instance, while autoregressive decoders scale efficiently with a balanced token-to-parameter ratio, transformer encoders (like BERT) operate under different constraints. Megi Dervishi et al. showed in Training compute-optimal transformer encoder models that encoders require a data-to-parameter ratio that is one to two orders of magnitude higher than decoders to avoid being under-optimized. When repurposing these models for contrastive tasks, such as text embeddings, Albert Q. Jiang et al. demonstrated in Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe that maximizing model size under a tight budget can backfire by restricting the volume of training data the model can process.
Sparsity introduces another dimension to this trade-off. In a Mixture of Experts (MoE) architecture, the total parameter count can be scaled up without a linear increase in compute per token. Elias Frantar et al. established in Scaling Laws for Sparsely-Connected Foundation Models that optimal weight sparsity increases alongside the training budget. Furthermore, Jan Ludziejewski provided a unified framework in Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient, demonstrating that when optimized correctly under a fixed parameter budget, sparse architectures can achieve superior data efficiency compared to dense alternatives.
Shifting the Frontier: Inference Costs and Data Constraints
Standard compute-optimal scaling laws focus exclusively on training compute. But what happens when a model is deployed to millions of users?
Nikhil Sardana addressed this in Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws, showing that for high-demand applications, the total lifecycle cost dictates "over-training" smaller models far past the Chinchilla optimal point. Spending more training compute to squeeze higher performance out of a smaller parameter footprint drastically reduces downstream inference costs.
Alternatively, Zhuohan Li et al. proposed a counterintuitive alternative in Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers: training a massive model, stopping early, and heavily compressing it. This approach achieves high accuracy within tight inference budgets while bypassing the long training times of smaller models.
Finally, when unique data is scarce, practitioners must trade off model width against training repetitions. Licong Lin et al. analyzed this in Improved Scaling Laws in Linear Regression via Data Reuse, proving that multi-pass training can achieve better error scaling rates than single-pass training up to a specific threshold, offering a compute-optimal path even when constrained by data scarcity.