The Depth to Width Ratio in Finite Neural Networks

This direction is split into other directions: stochastic kernel initialization. Its last version stays readable and citable here.

Version 1 of 119 papers35 conceptsfrozen 2026-10-04

In the theoretical study of deep learning, the "infinite-width limit" has long served as a powerful simplifying lens. By assuming a network’s layers are infinitely wide, researchers like Boris Hanin note that complex, non-linear models collapse into simpler linear systems governed by the Neural Tangent Kernel (NTK). In this idealized regime, the network is "lazy": its internal features barely change during training, and its behavior is perfectly deterministic.

However, real-world networks are finite. A growing body of research suggests that the validity of these infinite-width approximations depends not on width alone, but on the delicate balance between depth ($L$) and width ($W$). When a network becomes deep relative to its width, the elegant certainties of infinite-width theory begin to unravel.

The Critical Ratio

The transition from a predictable linear model to a chaotic, finite-width system is governed by the ratio of depth to width. Boris Hanin and David Rolnick demonstrated in How to Start Training: The Effect of Initialization and Architecture that the stability of a ReLU network is determined by the sum of the reciprocals of its layer widths ($\sum 1/w_l$).

If this value is too high—meaning the network is too deep for its width—the variance of activations and gradients doesn't just grow; it explodes or vanishes exponentially. This provides a rigorous mathematical explanation for the Exploding and Vanishing Gradient Problem (EVGP). As Boris Hanin proves in Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?, gradient stability is tied to this single architectural parameter, regardless of how carefully you scale your initial weights.

From Deterministic to Stochastic Kernels

In the infinite-width limit, the NTK is a fixed, deterministic matrix. But as the $L/W$ ratio increases, the kernel itself becomes a random variable. Mariia Seleznova and Gitta Kutyniok show in Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory? that in "chaotic" regimes, the variance of the NTK scales exponentially with the depth-to-width ratio.

This means that for deep, narrow networks, the "lazy training" assumption fails. The kernel shifts significantly during training, and the network's evolution becomes unpredictable. Zhengmian Hu and Heng Huang further refined this in On the Random Conjugate Kernel and Neural Tangent Kernel, proving that these kernels follow heavy-tailed log-normal distributions. The "randomness coefficient" of the network is directly proportional to $L/W$; as this ratio grows, the representations in the network become increasingly unstable.

Why ResNets Survive

If depth is so dangerous, why do we successfully train architectures with hundreds of layers? The answer lies in how different architectures handle the $L/W$ constraint.

Standard feedforward networks accumulate randomness at every layer, leading to the "hypoactivation" and exponential variance growth described by Mufan Li and Daniel M. Roy in The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization. Residual connections, however, act as a stabilizer. Hu and Huang demonstrate that ResNets maintain a bounded randomness coefficient even as depth approaches infinity, effectively decoupling the depth from the instability that plagues narrow feedforward chains.

Implications for Optimization

The breakdown of infinite-width theory isn't just a theoretical curiosity; it changes how we must optimize models. Antonio Orvieto and colleagues found in Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks that when width doesn't scale linearly with depth, the "curvature" of the loss landscape vanishes.

This creates flat, saddle-like plateaus where standard Stochastic Gradient Descent (SGD) gets stuck. Interestingly, this explains the empirical success of adaptive optimizers like RMSprop and Adam: they naturally rescale their steps to navigate the vanishing curvature caused by high $L/W$ ratios, succeeding where simpler methods fail.

Ultimately, this research direction suggests that width is the "budget" that allows depth to function. To maintain the stable, predictable learning dynamics promised by theory, a network's width must grow in tandem with its depth to keep the $L/W$ ratio—and the resulting stochasticity—under control.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_depth_to_width_ratio_in_finite_neura_5b1ecbe1,
  title = {The Depth to Width Ratio in Finite Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-depth-to-width-ratio-in-finite-neural-networks/rdr_5b1ecbe138a44ec4aa19ba170c63268c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_depth_to_width_ratio_in_finite_neura_5b1ecbe1_v1,
  title = {The Depth to Width Ratio in Finite Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-depth-to-width-ratio-in-finite-neural-networks/rdr_5b1ecbe138a44ec4aa19ba170c63268c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}