Mean Field Theory for Infinite Width Neural Networks

29 concepts28 papers1 proposal

Deep learning often operates in a regime where the number of parameters vastly exceeds the number of training samples. While classical statistics might predict catastrophic overfitting, these overparameterized models generalize remarkably well. To understand why, researchers have turned to the "infinite-width limit," where the number of neurons in a layer tends toward infinity.

A dominant framework for this analysis is Mean-Field Theory. Rather than tracking thousands of individual, discrete weights, the mean-field approach treats the collection of neurons as a continuous probability distribution or a "cloud of particles." This shift in perspective allows the complex, non-convex optimization of neural networks to be modeled as a smooth, predictable flow of a distribution.

From Discrete Weights to Continuous Measures

In a standard neural network, we update a finite set of weights $W$ using gradient descent. As the width of the network grows, the influence of any single neuron vanishes. Shevchenko et al. describe this limit by representing the hidden layer as a continuous distribution of neurons. By doing so, the training process is no longer a sequence of discrete steps in a high-dimensional weight space, but the evolution of a density function over time.

This abstraction transforms the study of neural networks into the study of Partial Differential Equations (PDEs). As noted by Calder et al., viewing discrete algorithms as "continuum limits" allows researchers to apply rigorous PDE theory to guarantee stability and convergence. In this view, neurons move like particles in a fluid, shifting their mass toward configurations that minimize the network's error.

Feature Learning vs. Lazy Training

A key contribution of mean-field theory is its ability to explain feature learning. For years, the "Neural Tangent Kernel" (NTK) regime was the primary tool for analyzing wide networks. However, as Misiakiewicz et al. and Mahankali et al. point out, the NTK regime describes "lazy training," where weights stay very close to their initial values and the network behaves like a fixed linear model.

Mean-field analysis describes the "rich" regime. By scaling weight updates differently—specifically, by the inverse of the number of neurons rather than the square root—the network can escape the linearized regime. This allows the internal features to evolve significantly from their initial state to adapt to the data structure (Zhang et al.; Yamamoto et al.). Neumayer et al. bridge these two worlds by introducing the "Scaling Path," showing how the scale of initialization determines whether a network performs lazy kernel learning or active feature learning.

Global Optimality in Non-Convex Landscapes

One of the most persistent mysteries in deep learning is why simple gradient descent finds global minima in highly non-convex landscapes. Mean-field theory provides a mathematical "guarantee of success." Fang et al. argue that overparameterization effectively "convexifies" the optimization landscape when viewed through the lens of measures.

By tracking the evolution of the weight distribution (often described as a Wasserstein flow), researchers have proven global convergence for various architectures:

  • Two-layer Networks: Chizat et al. and Akiyama et al. show that gradient descent on the distribution of neurons converges to a global optimum, even when the underlying landscape is non-convex.
  • Deep Residual Networks: Fang et al. extend this by tracking the distribution of features (outputs of hidden units) rather than weights, proving that overparameterized ResNets can escape lazy training.
  • Reinforcement Learning: Agazzi et al. and Leahy et al. apply mean-field limits to policy gradient methods, proving that entropy-regularized algorithms converge exponentially fast to optimal policies in continuous spaces.

Breaking the Curse of Dimensionality

Mean-field models also explain why neural networks outperform traditional kernel methods on complex tasks. Abbe et al. identify the "Merged-Staircase Property" as a condition that allows neural networks to learn sparse functions with far fewer samples than kernels require. Similarly, Zhang et al. prove that standard gradient descent allows networks to learn target functions with $O(d^{3.1})$ samples, while kernel methods are mathematically restricted to $d^4$.

This efficiency stems from the network's ability to adapt its features to the data's intrinsic geometry. Chen et al. and Domingo-Enrich et al. demonstrate that by training in the active mean-field regime, models can adapt to low-dimensional structures hidden within high-dimensional data, effectively breaking the curse of dimensionality.

Practical Implications: From $\mu$P to FNOs

The theoretical insights of mean-field limits are not just abstract; they inform how we train massive models today. Hong et al. provide a foundation for Maximal Update Parameterization ($\mu$P), a technique that allows researchers to tune hyperparameters on small models and transfer them directly to large ones by maintaining consistent infinite-width limits.

Furthermore, the mean-field perspective is expanding into specialized architectures. Fujisawa et al. use it to analyze Fourier Neural Operators (FNOs), tracking how feature covariance evolves through layers to solve the instability issues that often plague deep physics-informed models. By treating layers as continuous distributions, researchers can finally provide the "algorithmic justification" for why deeper, wider networks are not just more expressive, but more learnable (Ren et al.).

Key Foundational Work

Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory (2020) — contributed 1 concept elements to this direction.
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks (2020) — contributed 1 concept elements to this direction.
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_infinite_width_distribution_modeling_2c9757d1,
  title = {infinite-width distribution modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-distribution-modeling/rdr_2c9757d1c8ba4483bc0bb818ffcb24e4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_infinite_width_distribution_modeling_2c9757d1_v1,
  title = {infinite-width distribution modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-distribution-modeling/rdr_2c9757d1c8ba4483bc0bb818ffcb24e4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake