The Limits of Neural Scaling Laws

143 concepts119 papers1 proposal

For years, the development of large-scale models has been guided by a simple mantra: more compute, more data, and more parameters lead to predictably better performance. These "scaling laws" suggested a power-law relationship that showed no signs of stopping. However, as models have grown from millions to trillions of parameters, researchers are increasingly encountering a "sub-scaling" regime where these gains decelerate sharply or vanish entirely.

Understanding why scaling saturates is critical for determining whether the next order of magnitude in compute will yield a smarter model or simply a more expensive one.

The Transition to Sub-scaling

Traditional scaling laws assume that performance improves linearly in log-log space. However, Zhengyu Chen et al. in Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies argue that models eventually enter a sub-scaling phase. In this regime, the relationship between compute and performance becomes non-linear, and the returns on additional data diminish. This suggests that the "brute force" approach to intelligence is hitting a structural wall where the quality of data and the efficiency of training strategies become more important than sheer volume.

This saturation is not just a theoretical concern but a practical reality across various modalities. For instance, Ricard Marxer notes in Scaling Properties of Speech Language Models that speech models eventually reach a point where they have effectively "mastered" a task, causing compute requirements to grow much faster than power-law predictions as they approach these plateaus.

Domain-Specific Saturation Points

The point at which scaling yields diminishing returns varies significantly by task. While general language modeling may still show marginal gains at massive scales, specialized reasoning and technical tasks often plateau much earlier.

The Inverse Scaling Phenomenon

Perhaps most surprising is "inverse scaling," where larger models actually perform worse than their smaller counterparts. Oren Neumann et al. explore this in AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws, noting that regressions often occur when models focus on the wrong subsets of data as they scale.

This is particularly evident in high-stakes fields like medicine. Canyu Chen et al. demonstrate in ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction? that increasing parameter counts does not reliably improve clinical outcomes and can sometimes lead to worse performance. This suggests that for tasks requiring the interpretation of longitudinal patient records, scale alone cannot replace the precision of domain-specific modeling.

Why Scaling Fails to Bridge the Understanding Gap

The intuition behind these plateaus often points to a fundamental "understanding gap." Jared Moore argues in Language Models Understand Us, Poorly that the current scaling paradigm is hitting an economic and physical wall. Achieving human-level comprehension through pure scaling is projected to require investments exceeding one billion dollars per model, yet it may still fail to produce the grounded internal representations humans possess.

Furthermore, gains on static benchmarks do not always translate to real-world utility. Hussein Mozannar et al. found that significant leaps in benchmark accuracy for coding models often result in only minor improvements in actual human productivity in The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers.

As the industry reaches these limits, the focus is shifting from "more" to "better." The authors of EvoLM: In Search of Lost Training Dynamics for Language Model Reasoning suggest that excessive pre-training beyond a specific token-to-parameter ratio can actually degrade reasoning. This implies that the future of AI may lie not in larger models, but in optimizing the training dynamics and architectural efficiency of the models we already have.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_diminishing_returns_of_scale_48f64b07,
  title = {diminishing returns of scale},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diminishing-returns-of-scale/rdr_48f64b07df8c4a62aac527f7ca787e2e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_diminishing_returns_of_scale_48f64b07_v1,
  title = {diminishing returns of scale},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diminishing-returns-of-scale/rdr_48f64b07df8c4a62aac527f7ca787e2e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake