The Infinite Width Approximation Gap in Deep Learning

614 concepts451 papers1 proposal

To understand why a deep neural network generalizes so well, theorists often look at an idealized version of the model: one where the hidden layers are infinitely wide. In this infinite-width limit, the complex, chaotic training dynamics of a neural network simplify into a beautiful mathematical regime known as the Neural Tangent Kernel (NTK).

Under the standard NTK regime, an infinitely wide network behaves exactly like a linear model governed by a static, unchanging kernel. Optimization becomes convex, global convergence can be formally proven, and the network's behavior becomes completely predictable.

But this mathematical elegance comes with a catch. To make the math work, the infinite-width limit relies on an assumption called "lazy training." In an infinitely wide network, individual weights only need to move an infinitesimal amount from their random starting positions to achieve zero training error. Because the weights barely move, the network never actually learns new representations or features; it simply performs kernel regression using its initial, random features.

This creates what researchers call the infinite-width approximation gap. Real-world neural networks are finite, and they do not train lazily. They learn by fundamentally reshaping their internal representations.

How Finite Networks Deviate from Static Kernels

In practical, finite-width architectures, the static kernel assumption quickly falls apart. Instead of remaining frozen, the network's tangent kernel evolves dynamically throughout training.

Why does this happen? In a finite network, gradient descent forces parameters to undergo significant displacement from their initial values to minimize loss, as noted by Sun et al. and Hu et al.. As these weights move, the network's effective kernel changes. Doan et al. and Sihag et al. demonstrate that this empirical kernel evolution is continuous and highly dependent on the training data, meaning that predictions derived from a static infinite-width kernel fail to capture real training trajectories.

Standard NTK Regime (Infinite Width):
Initial Weights ───(Lazy Training)───> Weights Barely Move ───> Static Kernel (No Feature Learning)

Real-World Regime (Finite Width):
Initial Weights ───(Active Training)───> Significant Displacement ───> Evolving Kernel (Active Feature Learning)

This deviation becomes more severe depending on the geometry of the network. Hanin et al. proved that when a network is both deep ($d$) and wide ($n$), the NTK is actually stochastic at initialization, and its fluctuations away from the idealized infinite-width kernel grow exponentially with the depth-to-width ratio ($d/n$). When networks operate in highly non-linear regimes, this local linear approximation completely loses its accuracy, as shown by Halvagal.

The Non-Linear Advantage and Scaling Laws

The breakdown of the infinite-width approximation isn't just a theoretical nuisance—it represents a massive performance discrepancy. Real, finite-width neural networks routinely outperform their infinite-width kernel counterparts.

By analyzing empirical scaling laws, Vyas et al. revealed that finite-width networks possess significantly higher data-scaling exponents than static kernels. This means that as you increase the size of the training dataset, the error rate of a real neural network drops much faster than the error rate of its corresponding NTK model. Static kernels utilize information less efficiently because they cannot adapt their features to the data distribution.

This performance gap is driven by what Ortiz-Jimenez terms the "non-linear advantage." Rather than keeping its features fixed, a finite network actively rotates its kernel during optimization to align with the target function. This process of kernel rotation allows the network to compress irrelevant input dimensions and amplify important ones—a mechanism that static infinite-width theories are fundamentally blind to, as argued by Bordelon et al..

Structural Breakdowns of the Infinite Limit

Beyond feature learning, standard infinite-width approximations break down entirely when confronted with common architectural choices and initialization strategies used in production.

  • Architectural Bottlenecks: Modern networks rarely maintain uniform, massive width across all layers. Autoencoders rely on narrow bottlenecks, and many networks use a pyramidal structure where layers narrow progressively. Nguyen et al. demonstrated that the NTK framework becomes mathematically ill-defined if even a single layer in a deep network maintains a constant, finite width. Similarly, Joudaki et al. showed that the infinite-width assumptions of mean-field theory fail to capture how representations behave inside narrow bottleneck layers.
  • Chaotic Initialization: The stability of the infinite-width limit is highly sensitive to hyperparameters. Seleznova et al. showed that in "chaotic" initialization regimes—where variances are set such that gradients explode with depth—the underlying kernel fails to remain stationary, causing infinite-width optimization and generalization proofs to collapse.

Bridging the Gap via Feature-Learning Limits

Because the standard NTK limit fails to capture the core mechanism of deep learning—feature learning—theorists have developed alternative scaling frameworks to bridge this gap.

The most notable framework is the Maximal Update Parametrization ($\mu P$), introduced by Yang et al.. Traditional NTK scaling scales the weights in a way that forces the network into a lazy, linear regime as width approaches infinity. $\mu P$ rewrites these scaling rules, adjusting the learning rates and weight initializations dynamically with layer width.

By doing so, $\mu P$ ensures that even when the network becomes infinitely wide, its weights continue to undergo meaningful displacement, and its kernel continues to evolve. This allows researchers to study infinitely wide models that still retain the rich, non-linear feature learning dynamics of practical, finite-width networks.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_infinite_width_approximation_gap_1d92a2df,
  title = {infinite-width approximation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-approximation-gap/rdr_1d92a2df6798499ba26761691690b7f3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_infinite_width_approximation_gap_1d92a2df_v1,
  title = {infinite-width approximation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-approximation-gap/rdr_1d92a2df6798499ba26761691690b7f3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake