Scale-Dependent Intrinsic Dimension

82 concepts47 papers3 proposals

The Manifold Hypothesis is a cornerstone of modern machine learning: it posits that high-dimensional real-world data (like images, text embeddings, or physical measurements) actually concentrates around a much lower-dimensional, smooth geometric surface. Traditionally, algorithms attempt to find a single, fixed integer that represents this "true" intrinsic dimension (ID).

However, treating dimensionality as a static constant ignores a fundamental reality of data geometry: perceived dimensionality changes depending on the scale of observation. If you zoom in too close to a dataset, you might only see discrete quantization noise or high-dimensional stochastic fluctuations. Zoom out too far, and complex internal structures collapse into a single global trajectory or a single point.

To build reliable models, we must treat the intrinsic dimension not as a single number, but as a continuous function of scale.


The Intuition: The Geometry of a String

To understand why dimensionality depends on scale, imagine a tightly wound ball of string.

  • From across the room (macro-scale), the ball looks like a 0-dimensional point.
  • Moving closer (meso-scale), you see the trajectory of the string, which behaves like a 1-dimensional line.
  • Zooming in on the string itself (micro-scale), it resolves into a 3-dimensional cylinder.
  • Zooming in to the atomic level (sub-microscopic scale), the smooth cylinder dissolves into a cloud of 0-dimensional discrete particles.

As Kégl demonstrates in Intrinsic Dimension Estimation Using Packing Numbers, data manifolds exhibit this exact same behavior. High-dimensional spikes at microscopic scales typically represent noise, while a stable plateau across larger scales indicates the true dimensionality of the underlying manifold. A robust understanding of data requires tracking how structural characteristics transition across these resolutions.

[Micro Scale: Noise/Lattice] ---> [Meso Scale: True Manifold] ---> [Macro Scale: Global Trajectory]
   (High/Discrete Dim)                 (Stable Intrinsic Dim)              (Low/Compressed Dim)

Mechanisms for Probing Scale

How do we mathematically "zoom in" or "zoom out" on an abstract point cloud? Researchers have developed three primary mechanisms to introduce a scale parameter into dimensionality estimation.

1. Varying Neighborhood Sizes and Distance Ratios

The most direct way to alter scale is to change the size of the local neighborhood used to calculate dimension. Levina et al. et al. in Maximum Likelihood Estimation of Intrinsic Dimension modeled the distribution of distances to a point's nearest neighbors as a Poisson process, showing that averaging these local estimates across a range of neighborhood sizes is essential to smooth out small-scale perturbations.

However, changing the neighborhood size can be computationally expensive or require data decimation. To circumvent this, Doimo et al. developed the Gride estimator in The generalized ratios intrinsic dimension estimator. Instead of changing the physical search radius, they adjusted the indices of the nearest neighbors used in distance ratios (e.g., comparing the distance of the 10th vs. 20th neighbor rather than the 1st vs. 2nd). This allows the estimator to look at the manifold at coarser scales while retaining the full statistical power of the entire dataset.

In applied settings, like tracking material failures, Zhou et al. show in On local intrinsic dimensionality of deformation in complex materials that tuning these neighbor counts acts as a multi-scale filter—smaller counts isolate fine-scale microbands of particle deformation, while larger counts capture system-spanning global shearbands.

2. Controlled Noise Injection

Instead of changing how far we look, we can change how much detail is visible by deliberately blurring the dataset with varying levels of Gaussian noise.

Tempczyk introduced this via LIDL in LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood. By adding noise of variance $\sigma^2$ and using neural density estimators to track how the data's log-likelihood responds, $\sigma$ becomes a tunable scale parameter. If a dimension's variance is smaller than the injected noise, the estimator effectively ignores it. This allows the model to filter out thin, irrelevant dimensions or quantization artifacts.

Similarly, Horvat et al. in Intrinsic dimensionality estimation using Normalizing Flows trained normalizing flows on data augmented with different noise levels. By observing the response of the transformation's singular values, they mapped out a signature of the data's intrinsic structure, identifying exactly when specific dimensions lose their signal and dissolve into noise.

3. Systematic Subsampling

A third approach alters scale by changing the density of the data itself. Facco demonstrated in Estimating the intrinsic dimension of datasets by a minimal neighborhood information that systematically subsampling a dataset to smaller sizes effectively "zooms out" the neighborhood distances. By tracking how the minimal neighborhood information changes across these subsampled scales, researchers can isolate a stable plateau that represents the true manifold dimension while filtering out high-dimensional noise. Hein et al. utilized a similar framework in Intrinsic dimensionality estimation of submanifolds in Rd by fitting log-log plots of density statistics against sample sizes to find the scale where the absolute slope stabilizes.


Multiscale Frameworks in Practice

When we embrace scale-dependence, we can extract structural insights that single-scale methods completely miss.

Metric Space Magnitude

To evaluate the diversity of generative AI models, Limbeck et al. leveraged Metric Space Magnitude in Metric Space Magnitude for Evaluating the Diversity of Latent Representations. By modeling the "effective number of points" as a continuous function of a scaling parameter, they capture the hierarchical nature of embeddings. At low scales, the metric sees a single unified cluster; at high scales, it counts every unique point. This continuous curve allows for the detection of subtle representation issues like mode collapse and latent curvature.

Discrete and Fractal Spaces

In discrete domains—such as DNA sequences or categorical surveys—data is restricted to a grid, causing traditional geometric tools to fail. Macocco et al. developed the I3D algorithm in Intrinsic dimension estimation for discrete metrics. By adjusting search radii on lattices, they show how the estimated dimension transitions smoothly from local grid constraints to global fractal patterns, proving that discrete biological data often lives on surprisingly low-dimensional continuous manifolds.


When the Manifold Hypothesis Fails

Perhaps the most compelling reason to use scale-dependent analysis is that it reveals exactly where our assumptions about data break down.

In Token Embeddings Violate the Manifold Hypothesis, Robinson et al. tested whether Large Language Model (LLM) token spaces behave like smooth manifolds. By evaluating local dimension stability across expanding spatial scales, they discovered that the perceived dimension increases dramatically as the search radius grows. This scale-driven expansion proves that token spaces are not smooth manifolds at all; instead, they are riddled with geometric singularities, cusps, and jagged edges that contribute to model instability.

This phenomenon mirrors discoveries in fundamental physics. Giasemidis et al. proved in Dynamical dimensional reduction in toy models of 4D causal quantum gravity that space-time itself undergoes a scale-dependent transformation. At macroscopic scales, the universe exhibits its expected 4-dimensional structure, but due to connectivity constraints at the microscopic Planck scale, it dynamically reduces to a 2-dimensional manifold.

Whether analyzing the fabric of the universe, the latent spaces of AI, or complex materials, treating dimensionality as a scale-dependent function transforms a single static metric into a rich, multi-scale map of geometric structure.

Key Foundational Work

Intrinsic Dimension Estimation Using Packing Numbers (2002) — contributed 4 concept elements to this direction.
Maximum Likelihood Estimation of Intrinsic Dimension (2004) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 49 papers
  2. Version 1 · 2026-10-04 · 74 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_scale_dependent_dimensionality_00f11aaa,
  title = {scale-dependent dimensionality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-dimensionality/rdr_00f11aaabe6744d2837c8a05cb622b5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_scale_dependent_dimensionality_00f11aaa_v2,
  title = {scale-dependent dimensionality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-dimensionality/rdr_00f11aaabe6744d2837c8a05cb622b5f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake