Extrapolating Neural Scaling Laws to Frontier Models

60 concepts58 papers1 proposal

Neural scaling laws have served as the predictive compass of modern deep learning. The core intuition is elegant: by training a series of small, inexpensive proxy models (ranging from millions to a few billion parameters), researchers can fit a power-law curve that projects how a model will perform when scaled up by multiple orders of magnitude. This mathematical predictability allows organizations to commit millions of dollars in compute infrastructure with high confidence.

However, as the frontier of AI pushes toward trillion-parameter architectures, tens of trillions of tokens, and complex multi-modal inputs, this clean predictability begins to fracture. Extrapolating scaling laws across vast compute frontiers introduces significant risks, driven by the fragility of power-law fitting, non-linear hyperparameter dynamics, and unexpected regime shifts in the underlying data and optimization landscapes.

The Fragility of the Power-Law Fit

On a log-log plot, a power law appears as a straight line. This geometric simplicity is deceptive. Because we fit the line using data from small-scale pilot runs, even microscopic variations in the estimated slope or intercept can lead to wildly divergent performance projections when extrapolated across several orders of magnitude.

In (Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning, Li et al. demonstrate that seemingly minor, often-unreported choices in data aggregation, outlier exclusion, and optimization objectives can drastically alter frontier predictions. Without a rigorous consensus on safe extrapolation ranges, relying blindly on these fits introduces immense financial and computational risk.

To combat this instability, researchers employ "backtesting"—fitting parameters on ultra-small models to see if they accurately retrodict the performance of intermediate, already-trained models. As Schaeffer et al. show in Pretraining Scaling Laws for Generative Evaluations of Language Models, validating functional forms through rigorous backtesting is essential to ensure that a scaling law maintains its predictive stability across five or more orders of magnitude of compute.

Non-Linear Hyperparameter Dynamics

A common pitfall in scaling extrapolation is assuming that the optimal hyperparameters discovered for a small model remain static as the model grows. In reality, the internal optimization dynamics of neural networks shift fundamentally with scale.

Width and Noise Scale

As networks grow wider, their interaction with stochastic gradient descent changes. Park et al. demonstrate in The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study that wider networks require a higher normalized noise scale—achieved through increased learning rates or smaller batch sizes—to achieve peak generalization. If the training process isn't sufficiently "shaken" with noise as width scales, the expected generalization gains of larger models fail to materialize.

Token Horizons and Learning Rates

The optimal learning rate is not a fixed constant; it is tightly bound to the total duration of training. The authors of Scaling Optimal LR Across Token Horizons reveal that the optimal learning rate decreases predictably as the token horizon extends.

[ Small Token Horizon ] ---> High Optimal Learning Rate
[ Large Token Horizon ] ---> Low Optimal Learning Rate

Failing to adjust the learning rate schedule for massive datasets means a frontier model will operate under suboptimal optimization dynamics, causing it to underperform its power-law projections. To bypass the immense cost of tuning these dynamics at scale, Milsom et al. introduce "function-space learning rates" in Function-Space Learning Rates. By measuring how much a network's actual outputs change per update in a tiny proxy model, practitioners can seamlessly transfer optimal hyperparameter configurations across width, depth, and initialization scales without overhead.

Regime Shifts and the Breakdown of Power Laws

True extrapolation failure occurs when a model enters a new physical or statistical regime where the initial power-law assumptions no longer apply.

The Data Scarcity Regime

Standard scaling laws assume an abundant supply of high-quality data. When data is severely constrained, larger models overfit rapidly and can actually perform worse than smaller ones. To map this behavior, Wang et al. propose a "Rectified Scaling Law" in Data Efficient Neural Scaling Law via Model Reusing. They demonstrate that initializing large models with weights from smaller, pre-trained models ("model reusing") fundamentally alters the optimization trajectory, restoring the power law even under extreme data scarcity.

The Undertraining Paradox in Quantization

A similar regime shift occurs when compressing models via low-bit quantization. The authors of Low-Bit Quantization Favors Undertrained LLMs uncover a striking paradox: low-bit quantization (such as 2-bit or 3-bit) works remarkably well on models that have not been fully trained relative to their parameter size. However, as models are trained on tens of trillions of tokens and approach full convergence, their internal parameter representations become highly sensitive, leading to catastrophic performance collapse under aggressive quantization.

The predictive power laws governing quantization robustness at a small token scale completely break down as we move toward fully converged frontier models.

Bridging Pre-training Loss and Task Performance

Standard scaling laws focus almost exclusively on cross-entropy loss—a smooth, continuous scalar value. However, practitioners care about downstream task performance (e.g., a model's ability to solve a specific mathematical problem or generate functional code), which often appears to jump suddenly from zero to high proficiency. These sudden jumps are frequently labeled "emergent abilities."

In Predicting Emergent Abilities with Infinite Resolution Evaluation, Hu et al. argue that these discontinuities are often an artifact of low-resolution, non-linear evaluation metrics (like accuracy) rather than a fundamental shift in the model's internal capabilities. By introducing a high-resolution sampling metric called PASSUNTIL (massive sampling until success), they demonstrate that task performance actually scales along smooth, predictable trajectories that can be forecasted from tiny proxy models.

Low-Resolution Metric (Accuracy)  ---> Looks like a sudden step-function (Emergence)
High-Resolution Metric (PASSUNTIL) ---> Reveals a smooth, predictable power-law curve

To make these downstream predictions practical, Montgomery et al. develop context-aware scaling laws in Predicting Task Performance with Context-aware Scaling Laws. Their framework jointly models downstream performance as a function of both training compute and the inference-time context length, successfully capturing the "performance cliffs" that occur when a model's context capacity is saturated.

The Frontier Validation Gap

The ultimate challenge of scaling law research is the validation gap. While small-scale experiments allow us to formulate elegant mathematical relationships, verifying whether these laws hold true at the 70B+ or trillion-parameter scale remains prohibitively expensive.

Many critical dimensions of scaling remain unverified at the extreme frontier:

As Bian et al. emphasize in Scaling Inference-Efficient Language Models, future scaling laws cannot merely optimize for training efficiency. They must evolve to co-optimize model size, data composition, and architectural shape against the long-term computational costs of inference, ensuring that our mathematical extrapolations remain grounded in real-world utility.

Key Foundational Work

Scaling Effect of Self-Supervised Speech Models (2021) — contributed 1 concept elements to this direction.
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_law_extrapolation_limits_aefdad8b,
  title = {scaling law extrapolation limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-law-extrapolation-limits/rdr_aefdad8b165f426e8f3e1aadf0a041d7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_law_extrapolation_limits_aefdad8b_v1,
  title = {scaling law extrapolation limits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-law-extrapolation-limits/rdr_aefdad8b165f426e8f3e1aadf0a041d7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake