In classical statistics, high-dimensional learning is haunted by the "curse of dimensionality." If you have an input space with $d$ dimensions, the number of samples required to learn a smooth function typically grows exponentially with $d$. Yet, deep learning models routinely succeed on images with millions of pixels or text with massive vocabularies using relatively modest datasets.
The resolution to this paradox lies in the Manifold Hypothesis: the idea that real-world, high-dimensional data does not actually fill its ambient space. Instead, it concentrates on or near low-dimensional structures—manifolds—with far fewer degrees of freedom. A growing body of theoretical research demonstrates that neural networks are "manifold-adaptive," meaning their complexity and error rates scale with the data's intrinsic dimension ($d^*$) rather than its ambient dimension ($d$).
Bypassing the Curse of Dimensionality
The fundamental breakthrough in this area is the proof that neural networks can achieve approximation rates that ignore the "noise" of high-dimensional ambient space. Johannes Schmidt-Hieber showed that deep ReLU networks can approximate functions on a manifold at a rate determined solely by the intrinsic dimension $d^*$. This means that if you are classifying high-resolution images of faces, the network's difficulty is governed by the few variables that describe a face (pose, expression, lighting), not the total number of pixels in the image.
This isn't just a property of deep networks. Francis Bach demonstrated that even single-hidden-layer networks, when regularized by a specific "variation norm," can automatically adapt to low-dimensional structures. Similarly, Deanna Needell and colleagues have shown that even randomized models like Random Vector Functional Link (RVFL) networks can dodge the curse of dimensionality by leveraging the underlying geometric structure of the data.
The Geometry of Expressivity
How does a network physically "adapt" to a manifold? One intuitive mechanism is the creation of linear regions. ReLU networks partition the input space into a series of convex polytopes, where the network acts as a simple linear function within each polytope.
Razvan Pascanu, Guido Montúfar, and Yoshua Bengio proved that deep models are exponentially more efficient than shallow ones because they can "fold" the input space. Crucially, Saket Tiwari and George Konidaris found that for data on a manifold, the density of these linear regions is dictated by the manifold's intrinsic curvature and geometry. The network concentrates its expressive power exactly where the data lives, effectively ignoring the vast, empty regions of the high-dimensional ambient space.
Architectural Adaptivity
Different architectures exploit this geometric structure in unique ways:
- Convolutional Residual Networks (ConvResNets): Hao Liu, Minshuo Chen, and Wenjing Liao have shown that ConvResNets are particularly adept at adapting to the intrinsic geometry of Riemannian manifolds. Their sample complexity and depth scale according to the intrinsic dimension, explaining why they excel at high-resolution image tasks.
- Transformers: Recent work by Zhaiming Shen and Rongjie Lai suggests that Transformers function by "ignoring" high-dimensional noise. The required width and number of heads in a Transformer are determined by the manifold's intrinsic dimensionality, allowing them to remain effective even as the input sequence length or feature count grows.
- Feature Learning: Jonas Paccolat argues that networks actively squash uninformative dimensions. In the "feature learning" regime, the network fundamentally warps its internal geometry to align with the task, reducing the effective dimensionality of the problem as training progresses.
Generalization on Noisy Manifolds
In the real world, data rarely sits perfectly on a mathematically pure manifold. It is often "thickened" by noise. To address this, researchers like Uri Shaham and Alexander Cloninger have provided proofs showing that deep networks still approximate functions effectively even when data is only near a low-dimensional manifold.
A key concept here is the Effective Minkowski Dimension (EMD). This framework suggests that sample complexity depends on how the data is concentrated. As long as the data is primarily distributed within a lower-dimensional effective space, the task remains tractable. This explains why models generalize well even on "dirty" or noisy high-dimensional data: they are not learning the ambient space, but the underlying "shatterable" structure of the data Cloninger et al..
By shifting the focus from the number of input features to the intrinsic complexity of the data distribution, this research direction provides a rigorous mathematical foundation for the "black box" of deep learning's success. The models work because they are, by design and optimization, geometry-seeking engines.