In the ideal world of Neural Tangent Kernel (NTK) theory, an infinitely wide neural network ($n \to \infty$) behaves like a simple, deterministic linear model. At initialization, its kernel matrix—which dictates how inputs interact and how gradients flow—is a fixed, predictable constant. During training, this kernel remains "frozen," allowing researchers to perfectly predict the network's evolution using standard linear dynamics.
But real-world neural networks are not infinitely wide. They have a finite width $n$ and a substantial depth $d$. In this finite regime, the initial NTK is not a deterministic constant; it is a random variable. This shift from determinism to stochasticity fundamentally alters our understanding of deep learning dynamics.
The Infinite-Width Illusion
When we initialize a finite-width network, the weights are sampled randomly. In an infinitely wide layer, the law of large numbers averages out this randomness, yielding a deterministic kernel. However, as Littwin et al. demonstrate in On random kernels of residual architectures, the NTK of a finite-width network fluctuates randomly at initialization. These fluctuations introduce unpredictable training dynamics and instability.
This stochasticity becomes particularly acute when the network's width is scaled proportionally to the size of the training dataset. In this linear-width regime, Fan et al. show in Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks that kernel matrices exhibit high-dimensional stochasticity. Instead of converging to clean, predictable constants, they manifest complex, random spectral structures that resist standard fixed-kernel analysis.
The Depth-to-Width Ratio as an Amplifier
Why does finiteness matter so much for deep networks? The core insight is that randomness does not merely accumulate linearly layer by layer; it compounds multiplicatively. The critical metric governing this behavior is the depth-to-width ratio ($d/n$).
Standard NTK Limit: Width (n) ──> ∞, Depth (d) fixed ==> Deterministic, Frozen Kernel
Finite Deep Regime: Depth (d) / Width (n) > 0 ==> Exponentially Growing Variance
As proven by Hanin et al. in Finite Depth and Width Corrections to the Neural Tangent Kernel, the variance of the NTK at initialization scales exponentially with the ratio $d/n$. If a network is deep relative to its width, the initial kernel becomes wildly unstable. Bahri et al. corroborate this in Les Houches lectures on deep learning at large and infinite width, showing that the relative fluctuations of both the NTK and the squared entries of the input-output Jacobian scale exponentially with $d/n$.
This exponential scaling means that two networks sharing the exact same architecture, trained on the exact same data, can possess completely different kernels at the start of training simply due to different random seeds. One seed might yield a kernel that trains smoothly, while another might cause gradients to explode. As Seleznova et al. argue in Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?, average-case theoretical predictions become unreliable for individual model instances when the $d/n$ ratio is large. The deterministic assumptions of infinite-width theory dissolve because individual random fluctuations dominate the system.
Conversely, the kernel only converges to a predictable, stable limit when the width grows significantly faster than the depth ($d/n \to 0$), as explored by St-Arnaud et al. in Understanding the role of depth in the neural tangent kernel for overparameterized neural networks..
Chaos vs. Order at Initialization
The exponential explosion of kernel variance is heavily dictated by initialization hyperparameters, which place a network into either an "ordered" or a "chaotic" phase.
In Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization, Seleznova et al. map out these stability regimes:
- The Ordered Phase: The kernel remains stable, constant, and deterministic. This is the only regime where deep networks truly behave according to standard, static NTK theory.
- The Chaotic Phase: The kernel's variance explodes exponentially relative to $d/n$. In this regime, the kernel is highly random at initialization and evolves drastically after even minimal training steps, violating the frozen kernel assumption entirely.
Unless hyperparameters are explicitly tuned to remain within the ordered phase, deep finite networks exhibit extreme sensitivity to their initial random weights, causing them to deviate entirely from predicted linear behavior.
Taming the Variance
If deep networks naturally suffer from exponential kernel variance under standard initializations, why do deep models train so successfully in practice? Modern deep learning circumvents this breakdown through architectural design and alternative initialization strategies.
Residual Connections
One solution is altering the network topology. Littwin et al. demonstrate in On random kernels of residual architectures that residual and dense connections fundamentally alter how variance propagates. By providing skip pathways, ResNets and DenseNets maintain predictable, stable kernel dynamics at immense depths without requiring the massive, parameter-heavy explosion in width that vanilla architectures would need to suppress variance.
Orthogonal Initializations
Another approach is changing the statistical properties of the initial weights. Standard initializations use independent Gaussian distributions, which allow variance to compound exponentially across layers. In Feature learning and generalization in deep networks with orthogonal weights, Day et al. show that initializing weights with orthogonal matrices causes the fluctuations of the NTK to saturate after roughly 20 layers.
By preventing unbounded exponential growth, orthogonal weights decouple kernel stability from depth, ensuring stable training dynamics even when a network's depth is comparable to its width.
Key Foundational Work
Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory? (2021) — contributed 6 concept elements to this direction.
Finite Depth and Width Corrections to the Neural Tangent Kernel (2020) — contributed 2 concept elements to this direction.
On random kernels of residual architectures (2021) — contributed 1 concept elements to this direction.