Abstract
This paper establishes a theoretical link between Denoising Score Matching (DSM) loss and the Local Intrinsic Dimension (LID) of data manifolds. By proving that DSM loss serves as a lower bound for LID, the authors introduce a scalable, memory-efficient estimator that leverages pre-trained diffusion models. Empirical results demonstrate that this method outperforms existing state-of-the-art techniques in both accuracy and computational overhead across various high-dimensional datasets, including synthetic manifolds and latent representations of large-scale generative models.
Introduction
In machine learning, the manifold hypothesis suggests that high-dimensional data (such as high-resolution images) actually resides on much simpler, lower-dimensional surfaces. The number of variables needed to describe a point on this surface is its Local Intrinsic Dimension (LID). Knowing the LID is critical for anomaly detection, data compression, and understanding model generalization.
However, calculating LID for complex, high-dimensional data is notoriously difficult. Traditional non-parametric methods do not scale to modern datasets, and recent parametric methods using diffusion models are often computationally heavy, requiring dozens of model passes or expensive gradient computations. This paper seeks a "lighter," more scalable way to extract LID from generative models by leveraging the loss function used during training.
Theoretical Framework and Methods
The authors bridge the gap between Score Matching—the objective used to train diffusion models—and manifold geometry. Their primary contribution is theoretical, centered on Theorem 3.1 (The Denoising Bound).
The Denoising Bound
The authors prove that the DSM loss, which measures how well a model can remove noise from a sample, is lower-bounded by the LID ($d$). The intuition behind this bound lies in how a model handles noise relative to the data manifold.
As illustrated in the figure above, when noise is added to a point on a manifold, it can be decomposed into tangent and normal components. The model can easily learn to correct noise in the normal space (orthogonal to the manifold) because there is no data density there. Conversely, noise in the tangent space moves points along the manifold, making it harder for the model to distinguish between "noisy" and "clean" data. The resulting residual loss is thus proportional to the number of tangent directions, which defines the LID. By simply evaluating the DSM loss at a specific noise level $\sigma$, the authors create a "plug-and-play" LID estimator that requires no backpropagation.
Experimental Evaluation
The authors evaluated their method across a variety of settings, including benchmark manifolds from the scikit-dimension package (HyperSpheres, HyperBalls, Clifford Toruses) and real-world latent representations from Stable Diffusion 3.5 and Stable Diffusion 2.
Key Results
The DSM method demonstrates superior performance in both accuracy and resource utilization compared to existing baselines like FLIPD and ESS.
- Accuracy and Scalability: On benchmark manifolds, the DSM method achieved an average Mean Absolute Error (MAE) of 3.22, outperforming the previous state-of-the-art, FLIPD (MAE 4.32). While non-parametric methods like ESS work well on small problems, their error explodes as the ambient dimension increases. The DSM method remained stable and accurate even as dimensions scaled up.
- Memory Efficiency: Because the DSM estimator avoids backpropagation and gradient-heavy computations, it uses significantly less VRAM. In tests with Stable Diffusion 3.5, the DSM method used roughly 60% of the peak memory required by FLIPD.
- Robustness: The method is resilient to quantization noise, maintaining consistent estimates even when models are run in lower precision (e.g., float16), whereas gradient-based estimators are often sensitive to such noise.
Discussion and Future Work
While the DSM estimator provides a scalable solution, its accuracy is sensitive to the choice of noise scale ($\sigma$). If $\sigma$ is too large, the manifold structure is blurred; if it is too small, the model may only see individual data points. Future research will focus on automated "knee search" strategies for selecting optimal noise levels and accounting for high manifold curvature. Additionally, exploring the behavior of these estimators in distributed computing environments or with more aggressive quantization remains an open area for investigation.
Conclusion
By revealing that the training loss of diffusion models contains intrinsic geometric information, this work provides a powerful new lens for understanding data complexity. The DSM-based LID estimator offers a scalable, efficient, and accurate alternative for high-dimensional data analysis, proving that geometric signals are often "hidden in plain sight" within standard machine learning metrics.
Summary updated