Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, Luca Ambrogioni

Introduction

Diffusion models have emerged as a dominant paradigm in generative modeling, yet the underlying reasons for their superior performance over traditional likelihood-based methods—specifically their resilience to "manifold overfitting"—have remained largely empirical. This research, presented at ICLR 2025, provides a theoretical foundation for this success by analyzing the spectral properties of the score function's Jacobian through the lens of Random Matrix Theory (RMT) and statistical physics.

The Manifold Hypothesis and the Overfitting Problem

The manifold hypothesis posits that high-dimensional data, such as natural images, resides on low-dimensional surfaces (manifolds) embedded within a larger space. Traditional likelihood-based models, like VAEs or Normalizing Flows, often suffer from manifold overfitting. Because they attempt to maximize probability density on a zero-volume surface, the density technically diverges, causing models to become "blind" to the internal distribution of the data. They may learn the surface's location but fail to model how data is spread across it.

Spectral Analysis via Random Matrix Theory

The researchers address this by analyzing the Jacobian of the score function. In diffusion, the score is the gradient of the log-density, representing a vector field pointing toward the data. The Jacobian of this field reveals how the "pull" changes in different directions. Using RMT and the replica method, the authors derived the exact distribution of eigenvalues for this Jacobian. A "spectral gap"—a sharp drop in eigenvalues—indicates a clear separation between directions tangent to the manifold and those orthogonal to it.

Geometric interpretation of the spectral gap and the three phases of generative diffusion. (a) The eigenvalue spectrum reveals a gap between tangent and orthogonal spaces. (b) Visualization of these spaces on a manifold. (c) The three qualitative phases of the reverse diffusion process.

The Three Phases of the Generative Process

The core contribution of this work is the identification of three distinct geometric phases during the reverse (generative) diffusion process:

  1. Phase I: The Trivial Phase (Large $t$): At the start of generation, the process is dominated by noise. The Jacobian spectrum is flat, and the model does not yet "see" the manifold.
  2. Phase II: Manifold Coverage (Intermediate $t$): As generation progresses, spectral gaps begin to open. This is the critical "learning" phase where the model directs particles to spread across the manifold according to the data's internal density.
  3. Phase III: Manifold Consolidation (Small $t$): As generation nears completion, the score function acts primarily as a projection operator, "pinning" particles onto the data support.

This "division of labor" explains why diffusion models avoid overfitting: the internal distribution (Phase II) and the manifold geometry (Phase III) are handled at different points in time, preventing the final consolidation from erasing the learned internal density.

Empirical Results and Dimensionality Estimation

The theoretical predictions were validated using synthetic linear manifolds and natural image datasets including MNIST, CIFAR-10, and CelebA. The results demonstrate a remarkable match between derived RMT formulas and the numerical spectra of trained neural networks. Furthermore, the authors showed that the location of the spectral gap in Phase III can be used to estimate the intrinsic dimensionality of data; for instance, identifying a latent dimension of approximately 100 for MNIST.

Comparison of theoretical (blue) and numerical (orange) eigenvalue distributions across different generative time steps $t$. The evolution of the spectral gap confirms the transition through the three geometric phases.

Conclusion

By providing a rigorous geometric framework, this work clarifies why diffusion models succeed where others fail. While the current theory assumes local linearity, it successfully predicts the behavior of complex, non-linear models. Future work will look toward extending these RMT techniques to understand global topological properties and investigating specific neural network architectural artifacts that influence the Jacobian spectrum.

Summary updated

BibTeX
@inproceedings{ventura2025manifolds,
  author = {Enrico Ventura and Beatrice Achilli and Gianluigi Silvestri and Carlo Lucibello and Luca Ambrogioni},
  title = {{Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=KlN00vQEY2},
}
Figure