Imagine trying to calculate the density of a perfectly flat sheet of paper floating in a three-dimensional room. In a three-dimensional world, a perfectly two-dimensional surface has zero volume. If you attempt to calculate its probability density using standard formulas—where density equals probability mass divided by volume—you find yourself dividing by zero. The density blows up to infinity.
This mathematical crisis is the core challenge behind manifold overfitting and the intrinsic dimensionality mismatch in deep generative modeling. It stems from a fundamental conflict between two concepts:
- The Manifold Hypothesis: Real-world high-dimensional data (such as images, audio, or text) does not scatter uniformly across its embedding space. Instead, it concentrates on or near a much lower-dimensional sub-manifold.
- The Ambient Space Assumption: Standard probability density estimators (such as Normalizing Flows, Energy-Based Models, and Autoregressive models) assume that the data distribution possesses a well-defined density across the entire high-dimensional ambient space.
When a model built on the ambient space assumption tries to fit data restricted to a lower-dimensional manifold, the training dynamics destabilize, likelihoods diverge, and the model fails to capture the true underlying distribution.
The Pathology of Manifold Overfitting
Why exactly do standard models fail when encountering manifold data?
As formalised by Gabriel Loaiza-Ganem in Diagnosing and Fixing Manifold Overfitting in Deep Generative Models, a high-dimensional density model can achieve an arbitrarily high or even infinite likelihood by simply collapsing its variance to zero in the directions perpendicular to the data manifold.
Standard Density Model (Ambient Space) Manifold Overfitting
┌─────────────────────────┐ ┌─────────────────────────┐
│ . . . │ │ ║║║║║║║ │
│ . . │ ───────────> │ ║║ ║║ │
│ . . │ │ ║║ ║║ │
└─────────────────────────┘ └─────────────────────────┘
Spreads mass everywhere Collapses mass infinitely thin;
(Low likelihood, stable) (Infinite likelihood, unstable)
This behavior is a structural failure. The model maximizes its training objective by learning the geometric support—the "shape"—of the data, but it fails completely to learn the relative frequencies or probability mass distribution across that shape.
This dimensionality mismatch also breaks score-based and continuous-time generative models. As noted by Song et al. in Sliced Score Matching: A Scalable Approach to Density and Score Estimation, score matching techniques require the distribution to have support across the entire input space. When data is confined to a lower-dimensional manifold, the score function (the gradient of the log-density) becomes undefined or wildly unstable outside the manifold's support. This creates a severe gap between theoretical full-space assumptions and the geometric reality of natural data, a challenge analyzed in depth by George et al. in Analysis of Diffusion Models for Manifold Data and Tang et al. in Adaptivity of Diffusion Models to Manifold Structures.
To resolve this, researchers have developed three primary strategies: inflating the manifold with noise, altering the model's internal geometry, or abandoning ambient likelihood objectives altogether.
Strategy 1: Manifold Inflation via Noise
The most intuitive way to prevent densities from blowing up is to give the manifold some "thickness." By adding a controlled amount of noise to the data, we inflate the lower-dimensional manifold into a full-dimensional volume, ensuring the Jacobian determinants and likelihoods remain well-defined.
┌─────────────────────────┐ ┌─────────────────────────┐
│ ───────── │ ───────────> │ ░░░░░░░░░░░ │
│ │ Add Noise │ ░░░░░░░░░░░ │
└─────────────────────────┘ └─────────────────────────┘
Zero-volume Manifold Inflated Full-volume Manifold
(Infinite Density) (Finite, Stable Density)
Hyeongju Kim leveraged this concept in SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds by perturbing the data with random noise and training a normalizing flow conditioned explicitly on the noise level. This allows the model to capture thin structures during training while still being able to sample from the clean, zero-noise limit. Similarly, Horvat et al. developed Denoising Normalizing Flow, which optimizes a single joint objective that simultaneously inflates the data manifold and applies a denoising penalty to recover the underlying clean structure.
However, resolving manifold overfitting is not always as simple as adding noise. In Denoising Deep Generative Models, Loaiza-Ganem et al. demonstrated that even when noise is introduced to broaden the data distribution, generative models frequently maintain highly concentrated probability mass in narrow regions around the manifold. This persistent "peakiness" indicates that simple post-hoc denoising or naive noise addition is often insufficient to fully resolve the underlying dimensionality mismatch.
Strategy 2: Modifying the Model Geometry
Instead of altering the data to fit the model, a more robust approach alters the model's architecture to respect the data's intrinsic dimensionality.
Early non-parametric methods pioneered this direction. Vincent et al. introduced Manifold Parzen Windows, which replaced standard spherical kernels with "Gaussian pancakes" that dynamically align their principal axes to the local geometry of the data manifold, preventing probability leakage into empty regions of the ambient space.
In modern parametric modeling, researchers design transformations that explicitly account for dimensional changes:
- Rectangular Transformations: Standard normalizing flows require strictly square, invertible matrices to compute the change-of-variables formula. Edmond Cunningham bypassed this constraint in A Change of Variables Method For Rectangular Matrix-Vector Products by incorporating learned orthogonal noise, allowing flows to map low-dimensional latent spaces to high-dimensional outputs without restricting the density to a perfectly flat hyperplane.
- Varying Dimensionality: In Principal Component Flows, Edmond Cunningham forced latent variables to align with the data's principal manifolds, enabling density estimation on complex structures that exhibit varying dimensionality across different regions of the space.
- Velocity Field Decomposition: For continuous-time models like Flow Matching, Pi et al. proved in Learning Manifold Data with Flow Matching that decomposing the generative velocity field into a complex tangent component (moving along the manifold) and a simple linear normal component (moving toward the manifold) allows the model to escape the curse of dimensionality entirely. The learning complexity scales with the intrinsic manifold dimension rather than the massive ambient dimension.
Strategy 3: Bypassing Likelihood with Alternative Objectives
If absolute continuity and ambient density estimation are the root causes of mathematical instability, a final alternative is to abandon standard maximum likelihood estimation altogether. Instead, models can be trained using divergence measures that remain mathematically rigorous even when distributions do not share the same support.
Maximum Likelihood (KL) Optimal Transport / Stein
┌─────────────────────────────────────────┐ ┌─────────────────────────────────────────┐
│ Requires absolute continuity. │ │ Measures distance geometrically. │
│ Non-overlapping support = Undefined │ │ Non-overlapping support = Stable │
│ gradients or infinite spikes. │ │ gradients across empty space. │
└─────────────────────────────────────────┘ └─────────────────────────────────────────┘
For instance, Tolstikhin et al. utilized optimal transport in Wasserstein Auto-Encoders to provide stable, meaningful gradients even when the model's distribution and the true data distribution occupy non-overlapping manifolds.
Similarly, Fisher et al. introduced Measure Transport with Kernel Stein Discrepancy, leveraging Stein discrepancies to approximate target densities concentrated on low-dimensional sub-manifolds without requiring an absolute continuity assumption. In the context of energy-based modeling, Schröder et al. developed Training Discrete Energy-Based Models with Energy Discrepancy, bypassing the need for global density normalization and preventing contrastive objectives from failing when trying to distinguish a thin data manifold from the surrounding empty ambient space.
By shifting the focus from global ambient volume to local intrinsic geometry, these combined approaches allow modern generative models to successfully navigate the thin structures of real-world data without succumbing to numerical explosions or manifold overfitting.
Key Foundational Work
Classification via Minimum Incremental Coding Length (MICL) (2007) — contributed 1 concept elements to this direction.
Manifold Parzen Windows (2002) — contributed 1 concept elements to this direction.