The Universal Approximation Theorem is a cornerstone of deep learning, providing the mathematical guarantee that a neural network can represent any continuous function. Traditionally, this theorem focused on "wide" networks—those with a single hidden layer of potentially infinite neurons. However, as research shifted toward deep, narrow architectures, a new question emerged: how narrow can a network be before it loses its ability to learn?
Recent theoretical work has revealed that there is a "hard floor" for network width. If a hidden layer's width falls below a critical threshold, the network becomes topologically "choked," rendering it incapable of representing even simple functions, regardless of how many layers are added.
The Dimensionality of Folding
To understand why width is a bottleneck, we must view a neural network not just as a sequence of matrix multiplications, but as a continuous deformation of space. Activation functions like ReLU or Sigmoid are monotonic; they stretch and squash space but do not naturally "tear" it.
Boris Hanin and Mark Sellke demonstrated in Approximating Continuous Functions by ReLU Nets of Minimal Width that for a network to approximate any continuous function of $d_{in}$ variables, it must have a width of at least $d_{in} + 1$.
The intuition lies in the geometry of level sets—the regions of the input space where the network produces a constant output. To approximate a "bump function" (a function that is zero everywhere except for a small, isolated region), a network must be able to create "closed" or compact level sets. If the width is less than or equal to $d_{in}$, the network lacks the extra dimension needed to "fold" the input space back on itself. It is like trying to tie a knot in a piece of string while restricted to a flat 2D plane; without that third dimension to cross over, the string can never truly loop back.
The Closed Loop Obstruction
This topological restriction has practical consequences for classification. In Deep, Skinny Neural Networks are not Universal Approximators, Jesse Johnson proved that if a network's hidden layers are no wider than its input dimension, it cannot form bounded decision boundaries.
Imagine a dataset where one class is a circle in the center of a 2D plane, surrounded by another class. A "skinny" network (width $\le 2$) can only create decision boundaries that extend to infinity. It can slice the plane with lines, but it can never "pinch" a region to create a closed loop. To isolate that central cluster, the network requires at least one layer with $d_{in} + 1$ neurons to provide the ambient space necessary for the folding operation.
Untangling the Manifold
Beyond simple loops, width determines a network's ability to untangle complex data structures. Junyu Ren and Lek-heng Lim argue in Low-dimensional topology of deep neural networks that feedforward networks with monotonic activations are "topologically rigid."
In narrow settings, these networks cannot change the linking number of input manifolds. If two classes of data are intertwined like links in a chain, a narrow network cannot "cut and reglue" the space to separate them. This creates an expressivity ceiling: no matter how deep the network is, it remains a homeomorphism (a continuous deformation) that preserves the fundamental connectivity of the input. This explains why architectures like ResNets or Transformers, which use skip connections or attention to effectively "fold" or "jump" through space, are often more expressive than pure feedforward chains.
The Self-Intersection Problem
When we move from approximating values to approximating derivatives (the $C^1$ setting), the width requirements become even stricter. Geonho Hwang has shown that the minimum width is often dictated by the need to avoid self-intersections.
In Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach, Hwang uses tools from differential topology to show that for 2D-to-2D mappings, a width of 4 is often required to satisfy topological invariants like winding numbers. If the "pipe" of the hidden layers is too narrow, the mapping is forced to cross itself, creating singularities that prevent the network from accurately representing smooth, invertible functions.
Bottlenecks in Learning and Sets
These constraints aren't limited to standard MLPs. They appear in specialized architectures as well:
- Set Functions: Edward Wagstaff and colleagues found in On the Limitations of Representing Functions on Sets that the latent space dimension in "Deep Sets" architectures must be at least as large as the number of elements in the input set to maintain continuity.
- Training Dynamics: Marco Nurisso and Giovanni Petri demonstrated that "bottleneck" neurons—those with minimal connectivity—can trap training in disconnected regions of the parameter space, creating structural obstructions that no optimizer can bypass.
Summary of the Architectural Floor
The research across these papers suggests a unified view: Width is the dimension of the workspace.
While depth allows for the composition of functions and the creation of complex, "wiggly" boundaries, width provides the topological freedom to move, fold, and untangle the input space. If the width is too small ($W \le d_{in}$), the network is trapped by the topology of the input, unable to create the "islands" or "untied knots" necessary for universal approximation. As Yongqiang Cai notes in Achieve the Minimum Width of Neural Networks for Universal Approximation, the capacity of a network is fundamentally bounded by the geometry of its level sets. To learn truly complex data, a network must first have enough room to breathe.