The Universal Approximation Theorem is a cornerstone of neural network theory, yet it is famously misleading. It guarantees that a network with a single hidden layer can approximate any continuous function, but it says nothing about the efficiency of that representation. In practice, shallow networks are often mathematical "brute force" solutions, requiring an astronomical number of neurons to solve problems that deep networks handle with ease.
A growing body of theoretical research explains this discrepancy through the lens of scaling laws. The core insight is that for a vast class of "natural" functions—ranging from simple polynomials to complex solutions of partial differential equations (PDEs)—the resources required by a network (its depth and width) scale logarithmically with respect to the desired precision and the complexity of the task, provided the network is deep.
The Exponential Cost of Shallowness
To understand why depth is powerful, we must first see why shallowness fails. R. Srikant and Shiyu Liang demonstrate in Why Deep Neural Networks? that for smooth functions, shallow networks are wildly inefficient. To achieve the same level of accuracy as a deep network, a shallow architecture requires a number of neurons that grows exponentially with the input dimension.
This "curse of dimensionality" is a recurring theme. David Rolnick and Max Tegmark prove in The power of deeper networks for expressing natural functions that even for a simple multivariate polynomial—a product of $n$ variables—a shallow network needs $2^n$ neurons. In contrast, a deep network can represent the same function using a number of neurons that grows only linearly with $n$. By increasing depth logarithmically relative to the input dimension, we can maintain a manageable layer width while capturing immense complexity.
Logarithmic Scaling of Depth and Error
The most striking evidence for "depth supremacy" is how the approximation error $\epsilon$ behaves as we add layers. In many scenarios, the error does not just decrease; it decays exponentially with respect to depth.
Qingcan Wang shows that for analytic functions, deep networks achieve exponential convergence. This means that to reduce the error by a factor of 10, one only needs to add a constant number of layers, rather than multiplying the number of neurons. Consequently, the required depth $L$ scales as $L \propto \log(1/\epsilon)$.
This logarithmic relationship is not limited to abstract functions. Guido Montufar and Johannes Rauh found similar patterns in probabilistic models like Restricted Boltzmann Machines (RBMs), where the worst-case approximation error drops logarithmically as hidden units are added Expressive Power and Approximation Errors of Restricted Boltzmann Machines. Similarly, Guido Montufar and Jason Morton established that in Deep Belief Networks, the benefit of adding layers follows a logarithmic law, providing a formal mechanism for the trade-off between depth and accuracy Kernels and Submodels of Deep Belief Networks.
Overcoming the Curse of Dimensionality
Why does depth work so well for high-dimensional data, like images or physical simulations? The theory suggests that deep networks are uniquely suited to exploit the underlying structure of the data.
- Manifold Learning: Johannes Schmidt-Hieber proves that deep ReLU networks can approximate functions on low-dimensional manifolds at a rate determined by the manifold's intrinsic dimension rather than the high ambient dimension Deep ReLU network approximation of functions on a manifold.
- PDE Solvers: In the realm of physics, Dennis Elbrächter and Christoph Schwab demonstrate that deep networks can solve high-dimensional Kolmogorov PDEs without the curse of dimensionality DNN Expression Rate Analysis of High-Dimensional PDEs: Application to Option Pricing.
- Iterative Convergence: Koichi Taniguchi and colleagues suggest that the depth of a neural operator corresponds to the number of steps in an iterative solver (like Picard iteration), leading to exponential convergence in accuracy Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations.
Depth in Modern Architectures
This scaling logic extends to the most advanced models used today. Moritz Brösamle and Stephan Eckstein recently showed that Transformer decoders using Chain-of-Thought (CoT) reasoning rely on this same principle. To maintain computational expressivity under finite precision, the model's width and depth must scale logarithmically relative to the context length The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought.
Finally, Yuka Hashimoto and Sho Sonoda provide a "universal" proof for depth supremacy, showing that in most learning scenarios, the optimal depth scales logarithmically with the number of training samples Generalization Through Growth: Hidden Dynamics Controls Depth Dependence.
The intuition is clear: depth allows a network to build complex functions through hierarchical composition. Just as a few steps of a powerful algorithm can replace a massive lookup table, a few extra layers of a neural network can replace an exponential number of neurons. This logarithmic scaling is why deep learning is not just a preference, but a mathematical necessity for high-dimensional intelligence.