Neural Networks as Gaussian Processes

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2202 papers304 conceptsfrozen 2026-10-04

In deep learning, we usually think of a neural network as a complex, high-dimensional function defined by millions of weights. Training involves a messy, iterative search through this weight space to find a configuration that fits our data. However, a remarkable theoretical result simplifies this picture: as the width of a neural network approaches infinity, the network ceases to be a collection of weights and instead becomes a Gaussian Process (GP).

This equivalence allows us to treat deep networks as non-parametric Bayesian models, providing exact analytical solutions for inference and uncertainty quantification without ever running a single step of gradient descent.

The Central Limit Theorem of Neurons

The intuition behind this "infinite-width limit" begins with the Central Limit Theorem (CLT). Consider a single hidden layer in a neural network. Each neuron in that layer computes a weighted sum of its inputs followed by an activation function. If the weights are initialized independently from a zero-mean distribution, each neuron's output can be viewed as a random variable.

As Christopher K. I. Williams proved in Computing with Infinite Networks, when the number of these hidden units goes to infinity, the sum of their contributions to the output layer converges to a Gaussian distribution. Because this holds for any set of inputs, the entire network's output distribution over the input space becomes a Gaussian Process—a distribution over functions where any finite set of points follows a multivariate normal distribution.

From Shallow to Deep

For decades, this result was largely confined to single-layer networks. The breakthrough came when Jaehoon Lee, Yasaman Bahri, and their colleagues extended this to multi-layer architectures in Deep Neural Networks as Gaussian Processes.

They showed that the pre-activations of each layer in a deep network also converge to a Gaussian Process. This allows us to define a Neural Network Gaussian Process (NNGP) by recursively computing a "compositional kernel" that describes how covariance propagates through the layers. This kernel captures the architecture's inductive bias—how the network "thinks" two inputs are related—without needing to sample weights.

This framework has since been expanded to cover almost every modern architecture:

Why the Infinite Limit Matters

The primary advantage of the NNGP is that it provides exact Bayesian inference. In a standard finite network, we struggle with "uncertainty calibration"—knowing when the model doesn't know. By treating the network as a GP, we get a closed-form expression for the posterior variance, which serves as a mathematically grounded measure of uncertainty.

This has immediate practical applications:

  1. Neural Architecture Search (NAS): Instead of training thousands of candidate models, Jaehoon Lee showed in Towards NNGP-guided Neural Architecture Search that we can use the NNGP kernel at initialization to predict which architectures will perform best, speeding up the search by orders of magnitude.
  2. Robustness and Safety: Because GPs have well-defined kernels, researchers can calculate formal Robustness Guarantees for Bayesian Inference with Gaussian Processes, certifying that a model won't change its prediction under small adversarial perturbations.
  3. Efficient Ensembling: Moritz Akiya Zanger and colleagues demonstrated in Universal Value-Function Uncertainties that a single network can be used to distill the behavior of an infinite ensemble, providing robust uncertainty quantification in reinforcement learning without the cost of training multiple models.

The "Lazy" Learning Trade-off

While the infinite-width limit is mathematically elegant, it comes with a caveat: it often describes a "lazy" regime where the network's internal features do not change much during training. In this limit, the network behaves like a fixed kernel method.

To bridge the gap between theory and the "representation learning" we see in practice, researchers look at finite-width corrections. Devanshu Agrawal and Jacob Hinkle showed in Wide Neural Networks with Bottlenecks are Deep Gaussian Processes that adding a finite-width "bottleneck" layer prevents the network from becoming a simple GP, allowing it to compress data and learn more complex representations. Similarly, Jacob A Zavatone-Veth and Abdulkadir Canatar have derived formulas to characterize how finite-width Bayesian neural networks deviate from the static kernels of the infinite limit.

Ultimately, the Gaussian Process perspective provides the "North Star" for deep learning theory—a rigorous baseline that explains why wide networks generalize well and how we can harness their probabilistic nature.

Papers in this version

Showing 60 of 202 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 163 papers
  2. Version 1 · 2026-10-04 · 202 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_infinite_width_limit_behavior_c638f5d2,
  title = {infinite-width limit behavior},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/infinite-width-limit-behavior/rdr_c638f5d297f54114821977bc20ba9649},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_networks_as_gaussian_processes_c638f5d2_v1,
  title = {Neural Networks as Gaussian Processes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-networks-as-gaussian-processes/rdr_c638f5d297f54114821977bc20ba9649/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}