Deep neural networks are notoriously difficult to analyze. Their loss landscapes are non-convex, their parameters are high-dimensional, and their training dynamics are chaotic. However, a surprising mathematical simplification emerges when we take the "infinite-width limit." By imagining a network where the number of neurons in each hidden layer grows to infinity, the complex behavior of deep learning collapses into well-understood linear systems and stochastic processes.
The Network as a Gaussian Process
The first major insight into infinite-width networks involves their state at initialization. When a network is very wide, each neuron in a hidden layer receives a sum of many independent random signals from the previous layer. According to the Central Limit Theorem, the sum of many independent random variables tends toward a Gaussian distribution.
As Alexander G. D. G. Matthews and colleagues demonstrated in Gaussian Process Behaviour in Wide Deep Neural Networks, this means that an infinitely wide network at initialization behaves exactly like a Gaussian Process (GP). Instead of tracking millions of individual weights, we can describe the entire network's output using a simple covariance function, or "kernel," which determines how similar the network's predictions will be for two different inputs.
While early proofs were limited to simple feedforward structures, Greg Yang expanded this boundary in Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. By introducing a mathematical language called Tensor Programs, Yang proved that almost any modern architecture—including Transformers and RNNs—converges to a GP as it becomes infinitely wide.
Training as Linear Evolution
If wide networks are GPs at initialization, how do they behave during training? This is where the Neural Tangent Kernel (NTK) enters the frame.
In the infinite-width limit, the network becomes "lazy." Because there are so many parameters, each individual weight only needs to change by a tiny amount to significantly reduce the total error. As a result, the internal features of the network stay nearly constant, and the training process simplifies into a linear evolution. Sanjeev Arora, Simon Shaolei Du, and Zhiyuan Li showed in Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks that this NTK regime allows us to treat a deep network as a simple kernel regression model, which can even outperform finite-width models on small datasets.
However, this "lazy" behavior is a double-edged sword. While it makes the math elegant, it prevents the network from learning new features from the data—a core strength of deep learning. Boris Hanin noted in Finite Depth and Width Corrections to the Neural Tangent Kernel that real-world networks are finite, and their kernels actually "drift" during training, allowing for the feature learning that the standard NTK theory misses.
Recovering Feature Learning
To bridge the gap between "lazy" theoretical models and "active" practical networks, researchers developed new ways to scale parameters. Greg Yang and Edward J Hu introduced the Maximal Update Parametrization ($\mu P$) in Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks.
The intuition behind $\mu P$ is that by carefully adjusting the learning rates and initialization scales as the network gets wider, we can prevent the "lazy" collapse. This allows the infinite-width limit to preserve feature learning, providing a theoretical framework that actually matches the behavior of massive models like GPT-3. Similarly, Adam X. Yang and Maxime Robeyns proposed Deep Kernel Machines (DKMs), which use a Bayesian representation learning limit to maintain data-driven features even at infinite width.
The Frontier of Infinite Limits
The study of infinite-width limits has moved from a theoretical curiosity to a practical diagnostic tool. For example, Joshua DeOliveira and Walter Gerych demonstrated in The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training that NTK signals can be used to select architectures and clean data without ever actually training the model.
Despite these successes, significant challenges remain. Most rigorous proofs are still anchored to fully connected architectures. Researchers like Hyunouk Ko and Xiaoming Huo are working to extend these guarantees to universal consistency in Universal Consistency of Wide and Deep ReLU Neural Networks, while Feng-Lei Fan is exploring "infinite-depth" limits in Depth-induced NTK. The ultimate goal is a unified theory that explains why the specific structures of modern AI—like the attention heads in a Transformer—work so much better than the simple kernels of the past.