In machine learning theory, universal approximation theorems often tell us that shallow networks can represent almost any function or distribution we care about, provided they are wide enough. Yet, in practice, we almost always build deep networks. Why?
The traditional answer is that depth is more parameter-efficient. But a growing body of research reveals a more fundamental, practical reason: numerical stability. While a shallow network might theoretically be able to represent a complex target distribution, doing so often forces the network to become highly ill-conditioned, leading to exploding gradients and chaotic training dynamics. Stacking layers allows a network to break a violent, unstable transformation down into a sequence of smooth, well-conditioned steps.
The Illusion of Shallow Universality
To understand why depth is necessary for stability, we can look at normalizing flows—generative models that transform a simple base distribution (like a Gaussian) into a complex target distribution via a sequence of invertible mappings.
In theory, even simple shallow architectures like affine coupling networks are universal approximators. However, Frederic Koehler et al. demonstrated that this universality comes with a severe hidden cost. When a shallow network attempts to approximate a highly complex distribution (such as a mixture of Gaussians), its mathematical conditioning deteriorates dramatically. The network's layers must implement functions with extremely large Lipschitz constants and poor condition numbers.
In practice, an ill-conditioned layer acts like a magnifying glass for numerical errors. Small perturbations in the input lead to massive, unpredictable swings in the output, making the network nearly impossible to train using standard gradient-based optimization. Koehler et al. proved that this is not a failure of current training algorithms, but a fundamental geometric limitation: shallow invertible networks must trade away numerical stability to achieve high representational accuracy. Depth acts as the release valve for this tension, allowing the model to maintain a manageable Lipschitz constant across smoother, sequential transformations.
Depth-Induced Conditioning Beyond Flows
This relationship between depth and numerical conditioning is not unique to normalizing flows; it appears across radically different architectures. For instance, in reservoir computing, Deep Echo State Networks (DeepESNs) stack multiple recurrent reservoirs on top of each other.
When analyzing these systems, Alessio Micheli et al. discovered that increasing network depth naturally improves the condition number of the state matrix in higher layers. In a shallow ESN, the state matrix can easily become poorly conditioned, requiring complex, computationally expensive optimization techniques to train the linear readout layer. By stacking reservoirs, the authors showed that the deeper layers inherently develop more stable, well-conditioned dynamics. This depth-induced numerical conditioning allows practitioners to reliably train the network using simple, efficient optimization algorithms that would completely fail on shallower architectures.
The Parameter Bottleneck and the Dimensionality Wall
If depth provides stability by distributing the computational load, how exactly should we scale the number of layers relative to the complexity of our data?
The answer depends heavily on the capacity or "degrees of freedom" available within each individual layer. If each layer in a network is restricted to learning only a small number of parameters, the network must compensate by growing deeper.
Consider the task of "Gaussianization"—an iterative generative approach that transforms a distribution into a Gaussian using simple layers like random rotations and one-dimensional splines. Felix Draxler et al. analyzed the convergence rate of this method and established a strict lower bound on depth based on a parameter budget. Because a single random rotation layer only manipulates a linear number of parameters relative to the data dimensionality ($D$), it cannot capture the full quadratic degrees of freedom present in a target covariance matrix. Consequently, the authors proved that the required number of layers must scale at least linearly ($O(D)$) with the dimensionality of the data to guarantee convergence.
When layers are highly constrained, depth is the only mechanism available to accumulate the necessary parameters to represent the target distribution. This principle is mirrored in generative ReLU networks. Dmytro Perekrestenko et al. proved that a deep ReLU network can take a simple one-dimensional source variable and stretch, bend, and fold it to approximate any high-dimensional Lipschitz-continuous distribution.
Crucially, the authors showed that using a 1D source does not increase the approximation error compared to using a high-dimensional source, provided the network is sufficiently deep. The total number of connections required scales proportionally with the discrete complexity of the target distribution. Depth allows the network to efficiently budget its parameters, building up complex, high-dimensional structures through a long chain of simple, low-dimensional operations.
Conclusion
When designing deep generative models, it is tempting to view depth merely as a tool for increasing expressivity. However, the theoretical insights across these fields suggest a more nuanced reality:
- Shallow networks are trapped by instability: While technically capable of representing complex distributions, shallow architectures require extreme, ill-conditioned transformations that break optimization algorithms.
- Depth smooths the landscape: Distributing a transformation across many layers improves the condition number of the system, keeping Lipschitz constants bounded and training stable.
- Layer capacity dictates depth: The minimum depth of a network is fundamentally bounded by the ratio of the target distribution's complexity to the parameters available per layer.
Ultimately, depth is not just about what a network can represent in theory, but what can be safely and stably computed in practice.
Key Foundational Work
Richness of Deep Echo State Network Dynamics (2019) — contributed 2 concept elements to this direction.
Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks (2020) — contributed 1 concept elements to this direction.