Theoretical deep learning often relies on the "infinite-width limit," an abstraction where neural networks contain an unbounded number of neurons. In this idealized state, complex architectures simplify into predictable mathematical objects: randomly initialized networks become Gaussian Processes (GPs), and their training dynamics follow the Neural Tangent Kernel (NTK) or Mean-Field equations.
However, real-world networks are finite. The research direction explored here seeks to quantify the "speed limit" of this abstraction—exactly how fast a finite-width network approaches its infinite-width counterpart as neurons are added.
The Gaussian Speed Limit
At initialization, a wide neural network behaves like a Gaussian Process because each output is a sum of many independent random variables (the neurons). According to the Central Limit Theorem, such sums converge to a Gaussian distribution. But how much width is enough to trust this approximation?
Yang proved that almost all modern architectures, from Transformers to RNNs, converge to GPs, with the discrepancy diminishing at a rate proportional to $1/\sqrt{\text{width}}$. This $1/\sqrt{N}$ scaling is a recurring "speed limit" in the field. Basteri et al. provided explicit quantitative bounds using the Wasserstein distance, showing that to reduce the "non-Gaussianity" of a network by half, one must quadruple the hidden layer width.
However, this convergence isn't uniform across all settings. Eldan et al. noted that while smooth polynomial activations converge quickly, non-differentiable activations like ReLU converge much more slowly, creating a practical gap in how we model common architectures.
Dynamics and Propagation of Chaos
When we move from static initialization to active training, the neurons are no longer independent; they interact through the gradient descent signal. Theoretical models handle this using Mean-Field theory, treating the collection of neurons like a fluid flow or a probability distribution.
The accuracy of this "fluid" approximation depends on a concept called Propagation of Chaos. As De Bortoli et al. explained, this framework tracks how much the finite-width SGD noise deviates from the deterministic mean-field limit. They and Song et al. established that the number of neurons required for these approximations to hold is often independent of the input dimension, providing a "dimension-free" guarantee for high-dimensional learning.
Interestingly, the rate of convergence can vary depending on what you measure:
- Strong Error: The difference between individual weights in a finite vs. infinite network typically scales at $O(1/\sqrt{N})$.
- Weak Error: The difference in the expected output or loss can converge faster, at a rate of $O(1/N)$, as shown by Suzuki et al.. This suggests that even relatively narrow networks might mimic the performance of infinite ones more closely than their internal weight distributions would suggest.
The Cost of Lazy Training
A major tension in this research area is the gap between "lazy" and "rich" training regimes. In the lazy (NTK) regime, weights barely move from their initialization. Telgarsky et al. demonstrated a "tantalizing gap" here: while standard networks can approximate functions efficiently, their linearized NTK counterparts may require exponentially more neurons ($1/\delta^{10d}$ vs $1/\delta^{2d}$) to achieve the same accuracy.
Conversely, Zhang et al. argued that mean-field analysis proves finite-width networks can outperform kernel methods using significantly fewer samples. This suggests that the "rich" regime—where neurons actively learn features—is not just a theoretical curiosity but a source of representational efficiency. Pham et al. further showed that training actually "squeezes out" the randomness of initialization, leading to more stable and predictable trajectories in finite models over time.
Why These Bounds Matter
Quantifying these errors transforms theoretical physics-style "limits" into engineering "rulers." 1. Predictability: Agazzi et al. provided polynomial decay rates for trained networks, allowing researchers to estimate the accuracy loss of a finite model. 2. Optimization: For complex models like Variational Autoencoders, Wang et al. used finite-width NTK analysis to prove that reconstruction loss converges linearly, provided the width meets a specific threshold. 3. Generalization: Sreekumar et al. identified a "sweet spot" where network width is large enough to approximate a target (like KL divergence) but small enough to avoid overfitting, resulting in an optimal error rate of $O(n^{-1/4})$.
By establishing these non-asymptotic bounds, the authors across these papers bridge the gap between the "infinite" math of the classroom and the "finite" silicon of the data center.
Key Foundational Work
Mathematical Aspects of Neural Networks (2003) — contributed 1 concept elements to this direction.
The construction and approximation of neural networks operators with Gaussian activation function (2013) — contributed 1 concept elements to this direction.