Convergence of Infinite Width Networks to Gaussian Processes

275 concepts139 papers1 proposal

Deep neural networks are notoriously complex, high-dimensional, and non-linear. Yet, a surprising mathematical paradox emerges when we make them as large as possible: as the width of a neural network approaches infinity, its behavior simplifies dramatically.

Instead of wrestling with a chaotic distribution over millions of individual weights, the network's output distribution at initialization converges to a well-behaved, non-parametric Bayesian model known as a Gaussian Process (GP). This phenomenon, known as the Neural Network Gaussian Process (NNGP) limit, provides an elegant analytical lens to understand why deep networks generalize, how they represent data, and how they quantify uncertainty.

The Core Intuition: The Central Limit Theorem

To understand why an infinitely wide network becomes a Gaussian Process, consider a single hidden layer with $H$ hidden units. The output of the network for a given input is a weighted sum of the activations of these hidden units. If the weights and biases are initialized randomly and independently from standard distributions (such as Gaussians), each hidden unit acts as an independent random feature.

When we take the infinite-width limit ($H \to \infty$), we are summing up infinitely many independent, identically distributed random variables. By the Central Limit Theorem (CLT), the sum of these variables converges to a Gaussian distribution. Consequently, the joint distribution of the network's outputs for any finite set of inputs converges to a multivariate Gaussian distribution.

This foundational insight was first proven for single-layer networks by Christopher K. I. Williams in Computing with Infinite Networks. Instead of performing expensive, iterative training or sampling in parameter space, the NNGP limit allows practitioners to bypass individual weights entirely and perform exact analytical inference directly in function space.

Going Deeper: Recursive Kernels

What happens when we add multiple layers? In Gaussian Process Behaviour in Wide Deep Neural Networks, Jiri Hron extended this theory to deep fully connected architectures.

When every hidden layer becomes infinitely wide sequentially, the outputs of one layer serve as the inputs to the next. Because the pre-activations of each layer are Gaussian distributed due to the CLT, the covariance (or kernel) of the current layer can be computed deterministically by taking an expectation over the Gaussian distribution of the previous layer. This creates a recursive chain:

$$\mathcal{K}^{\ell}(x, x') = \mathbb{E}_{f \sim \mathcal{GP}(0, \mathcal{K}^{\ell-1})} \left[ \phi(f(x)) \phi(f(x')) \right]$$

where $\phi$ is the activation function. This means that a deep, infinite-width network can be exactly represented by a single, deterministic recursive kernel function.

Universality Across Architectures

The convergence to a Gaussian Process is not a unique quirk of simple fully connected networks; it is a universal property of overparameterization. By scaling the appropriate dimension to infinity, researchers have derived exact NNGP kernels for virtually every major architecture in deep learning:

  • Convolutional Networks: As the number of channels per layer approaches infinity, Garriga-Alonso et al. demonstrated that CNNs converge to GPs. Crucially, this infinite-width limit preserves spatial inductive biases like translation invariance.
  • Attention and Transformers: In Infinite attention: NNGP and NTK for deep attention networks, Hron et al. proved that while single-head attention behaves anomalously in the limit, multi-head attention architectures smoothly converge to GPs as the number of attention heads approaches infinity.
  • Graph Neural Networks: Niu et al. and Wu et al. applied this theory to graph-structured data, showing that infinitely wide graph convolutional networks converge to GPs with recursive kernels that explicitly encode graph topology.
  • Neural Operators: For models designed to learn mappings between infinite-dimensional function spaces (such as PDE solvers), de Souza et al. proved that an infinite number of hidden channels yields convergence to function-valued Gaussian processes.
  • Structural Constraints: This convergence is robust to architectural optimizations. Even when networks utilize hardware-efficient structures like block-sparse or low-rank factorized weight matrices, Nait Saada et al. proved that an exchangeable version of the CLT ensures they still converge to the exact same GP limits at initialization.

A unified mathematical justification for this universality was established in Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes, which introduced the framework of Tensor Programs to prove that any architecture composed of standard matrix multiplications and coordinate-wise non-linearities enters a GP regime at infinite width.

From Discrete Points to Continuous Function Space

To fully trust the NNGP approximation, we must ensure the network behaves smoothly across the entire continuous input domain, not just at a collection of discrete training points.

Bracale et al. addressed this by proving functional convergence for deep networks. By analyzing characteristic functions and exchangeable sequences, the authors showed that the entire neural network function converges weakly to a Gaussian Process indexed on path spaces. This ensures that global properties of the network—such as its maximum value over an interval or its integral—converge predictably, allowing researchers to treat wide networks as continuous stochastic processes.

Practical and Theoretical Insights

Why do we care that wide networks are secretly Gaussian Processes? This equivalence unlocks powerful tools for both analyzing and improving deep learning models.

Principled Uncertainty and Ensembles

Deep ensembles are highly effective at quantifying uncertainty, but they are computationally expensive and historically lacked strong Bayesian guarantees. Pearce et al. bridged this gap by introducing anchored ensembling—a method where individual networks are regularized toward random priors rather than zero. The authors proved that under infinite width, the predictive distribution of an anchored ensemble converges exactly to a GP posterior. This allows scalable deep ensembles to inherit the exact, non-parametric uncertainty intervals of Bayesian methods, a property utilized by Jiang et al. to provide reliable confidence intervals for multi-task causal effect estimation.

Understanding Simplicity Bias

Deep networks have the capacity to fit completely random labels, yet they generalize remarkably well on real-world data. The NNGP limit helps explain this mystery through the lens of architectural priors. De Palma et al. utilized the GP approximation to prove that random, wide ReLU networks are inherently biased toward "simple" functions. Because the prior probability space is naturally tilted away from highly oscillatory functions, the network resists overfitting even before training begins. Furthermore, De Palma et al. leveraged this geometric perspective to establish universal scaling laws for adversarial robustness, demonstrating how a random network's distance to a classification boundary scales predictably with its input dimension.

Beyond the Gaussian Limit

Despite its mathematical elegance, the standard NNGP limit has a notable limitation: Gaussian distributions are inherently "thin-tailed." This means they cannot easily model heavy-tailed data, sparse features, or complex inter-output correlations that real-world deep networks handle with ease.

To overcome this constraint, recent research extends the infinite-width framework beyond standard Gaussians:

  • Scale Mixtures: Hyungi Lee et al. introduced Scale Mixtures of NNGPs by placing a random scale prior on the network's readout layer. This simple modification allows infinite-width networks to represent a much broader class of stochastic processes, such as heavy-tailed Student’s $t$ processes, significantly improving robustness to outliers.
  • Infinitely Divisible Distributions: Hoil Lee et al. moved beyond independent Gaussian weights by modeling infinite-width limits using Lévy measures. This approach yields a mixture of Gaussian processes that successfully preserves heavy tails, sparsity, and compressibility in the infinite limit.

By treating the infinite-width limit not as a rigid boundary, but as a flexible foundation, these extensions allow researchers to preserve the analytical tractability of function-space inference while capturing the rich, non-Gaussian nuances of practical deep learning.

Key Foundational Work

Computing with Infinite Networks (1996) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 163 papers
  2. Version 1 · 2026-10-04 · 202 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_infinite_width_limit_behavior_c638f5d2,
  title = {infinite-width limit behavior},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-limit-behavior/rdr_c638f5d297f54114821977bc20ba9649},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_infinite_width_limit_behavior_c638f5d2_v2,
  title = {infinite-width limit behavior},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-limit-behavior/rdr_c638f5d297f54114821977bc20ba9649/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake