Classical statistical learning theory presents a paradox. If you try to learn an arbitrary function or probability distribution in a high-dimensional space, you immediately run into the curse of dimensionality. Non-parametric estimation theory dictates that the number of samples required to achieve a target error scales exponentially with the dimension—formulated as $\Omega(e^d)$ or sample complexities like $\Theta(n^{-\frac{2}{d+4}})$. For an image with a million pixels, the required dataset size would exceed the number of atoms in the observable universe.
Yet, deep neural networks routinely learn to classify images, generate audio, and model complex distributions using relatively modest datasets. This success implies that the curse of dimensionality is not an inescapable law of nature, but a symptom of overly pessimistic assumptions.
Recent theoretical work reveals that neural networks bypass this curse through two complementary mechanisms: exploiting the hidden low-dimensional structure of real-world data, and utilizing active feature learning to adapt their architectures to that structure.
The Manifold Hypothesis and Intrinsic Dimension
The foundational reason high-dimensional learning is possible is that real-world data rarely fills the ambient space uniformly. Images of faces, for instance, occupy a tiny, highly structured fraction of all possible pixel combinations.
As Wibisono et al. demonstrated in Optimal score estimation via empirical Bayes smoothing, if you make absolutely no structural assumptions about a distribution, estimating its score function (the gradient of the log-density) inherently suffers from the classical minimax rate of $\Theta(n^{-\frac{2}{d+4}})$. This proves that the curse is mathematically real for unstructured data.
To beat this rate, models must exploit the manifold hypothesis—the assumption that data concentrates near a low-dimensional surface embedded within the high-dimensional ambient space.
When this assumption holds, the statistical complexity of generative models shifts from depending on the ambient dimension to depending on the intrinsic dimension of the data manifold. In On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds, Dahal et al. proved that the approximation and estimation errors of deep generative models (such as GANs) are governed by this lower intrinsic dimension. Similarly, Schreuder et al. established risk bounds in Statistical guarantees for generative models without domination showing that smooth generators can accurately reconstruct distributions without needing a sample size that scales with the output space's dimensionality.
Ambient High-Dimensional Space (Dimension D)
┌────────────────────────────────────────────────────────┐
│ │
│ Data Manifold (Intrinsic Dimension d << D) │
│ ┌────────────────────────────────────┐ │
│ │ • Data Point │ │
│ │ • Data Point │ │
│ │ • Data Point │ │
│ └────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────┘
Neural networks restrict their learning to the manifold,
ignoring the vast, empty ambient space.
This mechanism explains the empirical success of diffusion models. Chen et al. proved in Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data that when data concentrates around a low-dimensional linear subspace, the sample complexity of score-based diffusion models scales according to the subspace's dimension rather than the ambient space.
But how does a network locate this subspace during training? Wang et al. provided an intuitive answer in Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering. By modeling data as a Mixture of Low-Rank Gaussians, the authors showed that training a diffusion model is mathematically equivalent to solving a subspace clustering problem. The network dynamically identifies the local low-dimensional structures, allowing it to generate high-resolution data efficiently.
Active Feature Learning vs. Static Kernels
Exploiting a low-dimensional structure requires an architecture capable of finding it. This is where neural networks fundamentally outperform classical non-parametric methods, such as kernel methods.
Kernel methods rely on fixed, static feature mappings. In When Do Neural Networks Outperform Kernel Methods?, Song Mei demonstrated that fixed-feature models require exponential sample sizes to approximate target functions unless the data's low-dimensional structure happens to align perfectly with the pre-defined kernel. If the structure is rotated or non-linear, the kernel method is blind to it and succumbs to the curse of dimensionality.
Neural networks escape this limitation through active feature learning. Instead of keeping features fixed, gradient descent actively rotates and warps the network's internal representations to align with the data's hidden coordinates.
Domingo-Enrich et al. analyzed this in the context of energy-based models in On Energy-Based Models with Overparametrized Shallow Neural Networks. They showed that when overparameterized shallow networks operate in the "active" feature-learning regime, they adapt directly to hidden low-dimensional structures. This self-adaptation collapses the effective dimensionality of the problem, yielding statistical convergence rates that bypass the ambient space's complexity.
Architectural Priors and Geometric Stability
A second way neural networks mitigate the curse of dimensionality is by embedding structural priors directly into their network architectures. These priors restrict the space of learnable functions to those that are physically meaningful.
Consider a natural image: if you shift it by a few pixels, or slightly deform its geometry, its semantic meaning remains unchanged. Generic statistical models treat a shifted image as an entirely new data point in high-dimensional space. Neural networks, particularly Convolutional Neural Networks (CNNs), do not.
In On the Sample Complexity of Learning under Geometric Stability, Bietti et al. proved that incorporating geometric priors—such as translation invariance and deformation stability—drastically reduces the sample complexity of learning. Instead of searching the entire high-dimensional space of all possible functions, the model only searches the space of stable functions.
This structural restriction dramatically reduces parameter and sample requirements:
- Weight Sharing and Locality: Ding-Xuan Zhou proved in Universality of Deep Convolutional Neural Networks that deep CNNs function as universal approximators while avoiding the exponential explosion of parameters seen in fully connected networks, precisely because they constrain learning to local spatial interactions.
- Eigenspace Restructuring: Lechao Xiao expanded on this mechanism in Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks, demonstrating that deep CNNs fundamentally reshape network learnability. By balancing spatial locality and interaction frequency, deep architectures restructure their eigenspaces during training, allowing them to learn complex high-dimensional functions with manageable amounts of data.
Achieving Dimension-Free Scaling
The ultimate realization of this research direction is the discovery that for certain models, sample complexity can become nearly dimension-free.
In Dimension-free Score Matching and Time Bootstrapping for Diffusion Models, Kumar et al. proved that diffusion models can learn high-dimensional distributions with a sample complexity that scales with $\log \log d$ rather than polynomials of $d$. They achieved this bound by analyzing how diffusion models learn across multiple noise levels simultaneously. By using a technique called Bootstrapped Score Matching, the model uses scores learned at cleaner levels to reduce variance at noisier levels, effectively decoupling the learning difficulty from the ambient dimension.
Complementing this, Pedrotti et al. demonstrated in Improved Convergence of Score-Based Diffusion Models via Prediction-Correction that predictor-corrector sampling algorithms maintain an efficiency that scales only logarithmically with data dimension.
These insights explain why deep learning does not stall as we move to higher-resolution data. The models are not learning the pixels; they are learning the underlying, low-dimensional geometric laws that govern how those pixels come together.