The Mechanistic Foundations of Neural Scaling Laws

157 concepts130 papers2 proposals

Deep learning has long been guided by a remarkable empirical regularity: as you scale up a model's parameters ($N$), dataset size ($D$), or compute budget ($C$), its test loss decreases predictably according to a power law ($L \propto N^{-\alpha}$). While these scaling laws have served as an engineering blueprint for building large language models, they have historically been treated as a black box. Why do these specific power-law exponents emerge? Why doesn't performance plateau abruptly, and what determines the slope of the improvement?

To move beyond empirical curve-fitting, researchers are developing a mechanistic scaling theory—a first-principles explanation that derives scaling laws from the interplay of data statistics, network geometry, and optimization dynamics.

Data Geometry: Zipfian Distributions and Hierarchies

One of the most intuitive explanations for neural scaling points to the structure of the data itself. Natural language, images, and code are not random; they are highly structured, heavy-tailed, and compositional.

Consider Zipf's law, which states that the frequency of a feature or word is inversely proportional to its rank. When a model trains on such data, it encounters a few highly frequent features very early, while rare features appear only as the dataset grows. In Scaling Laws for Associative Memories, Vivien Cabannes et al. modeled transformer layers as associative memories to bridge abstract mathematical theory and practical behavior. They proved that the heavy-tailed nature of language directly drives power-law generalization scaling, meaning the power-law exponent is a reflection of data frequency.

However, real-world data is more than just a flat list of features with varying frequencies; it is hierarchically composed (e.g., letters form words, words form sentences, sentences form concepts). Francesco Cagnetta et al. unified these perspectives in Learning curves theory for hierarchically compositional data with power-law distributed features. By analyzing a Random Hierarchy Model, the authors demonstrated a key distinction: while feature frequency dictates the scaling exponent in simple classification tasks, the underlying hierarchical structure is the primary driver of the scaling exponents observed in next-token prediction.

Feature Packing: Superposition and Sequential Learning

How do networks physically store these power-law distributed features within a finite hidden dimension? The answer lies in the geometry of high-dimensional spaces.

In Superposition Yields Robust Neural Scaling, Yizhou Liu et al. proposed that representation superposition—the ability of a neural network to pack more features into a hidden space than it has dimensions—is the primary driver of width scaling laws. As network width increases, the geometric capacity to pack features without destructive interference scales predictably, yielding the characteristic $1/m$ loss scaling seen in practice.

This smooth, global power law can seem paradoxical: shouldn't learning individual features cause discrete, jagged jumps in performance? Gerard Ben Arous et al. resolved this paradox in Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws. They demonstrated that the smooth decay in loss arises from the sequential recovery of individual signal directions during gradient descent. As the model scales, it uncovers independent signals one after another. When these discrete learning events are aggregated across a vast feature space, they smooth out into a continuous power law.

This aligns closely with the framework presented in The Quantization Model of Neural Scaling, where the authors argued that networks learn discrete "quanta" of skills or facts in order of their frequency in the data, explaining both the smooth power-law drop in overall loss and the sudden "emergence" of specific capabilities.

Architectural and Optimization Bottlenecks

Scaling laws are not solely a reflection of data; the mathematical operations inside the architecture and the dynamics of gradient descent impose their own fundamental limits.

For instance, why does loss often decelerate over time? Yizhou Liu et al. investigated this in Universal One-third Time Scaling in Learning Peaked Distributions. They found that the combination of the softmax function and cross-entropy loss creates an intrinsic optimization bottleneck when learning low-entropy, "peaked" distributions. This architectural interaction forces a universal $1/3$ time-scaling exponent, showing that some scaling limits are hardcoded into our choice of loss functions rather than data properties.

From an optimization perspective, Andrei Mircea et al. identified a specific bottleneck in Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning. As training progresses, gradients from different training examples begin to actively cancel each other out—a phenomenon they call Zero-Sum Learning (ZSL). The authors demonstrated that scaling up model parameters works primarily by delaying this bottleneck, giving the optimizer more geometric "room" to navigate competing signals without cancellation.

The Phase Diagram of Compute-Optimal Scaling

When we combine data complexity, network capacity, and optimization noise, we find that "optimal scaling" is not a static, singular rule. Instead, it behaves like a physical system undergoing phase transitions.

In 4+3 Phases of Compute-Optimal Neural Scaling Laws, Courtney Paquette et al. mapped this landscape by analyzing a solvable "Power-Law Random Features" model. They discovered that compute-optimal scaling is governed by a complex phase diagram consisting of four major phases.

Depending on the complexity of the data and the noise level of the optimizer, the optimal allocation of compute shifts:

  • In some phases, compute is best spent aggressively increasing model size ($N$).
  • In other phases, it is best spent training for more steps ($D$).

By framing scaling laws as emergent physical properties of high-dimensional feature packing and gradient dynamics, this mechanistic view allows us to see why scaling works—transforming an empirical observation into a predictable, engineering science.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 131 papers
  2. Version 1 · 2026-10-04 · 200 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_mechanistic_scaling_theory_27839d11,
  title = {mechanistic scaling theory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mechanistic-scaling-theory/rdr_27839d111a294f3d93115d40cabee452},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_mechanistic_scaling_theory_27839d11_v2,
  title = {mechanistic scaling theory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mechanistic-scaling-theory/rdr_27839d111a294f3d93115d40cabee452/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake