Modern machine learning operates in massive vector spaces. A text token, an image embedding, or a physical state simulation might easily live in a space with thousands of dimensions. However, this "ambient dimension" is highly deceptive. Data points rarely fill these vast voids uniformly; instead, they cluster along highly constrained, lower-dimensional geometric structures known as manifolds.
The true number of independent variables required to describe this underlying structure is its intrinsic dimension (ID). Understanding and estimating this value allows us to peer past the bloated representations of deep networks to see the true complexity of the information they process.
The Core Intuition: How Neighborhoods Scale
How do we measure the dimension of a space if we only have an unorganized cloud of data points? The key lies in a fundamental geometric property: how volume scales with radius.
Imagine a uniform distribution of points.
- In a 1D space (a line), doubling the length of a segment doubles the number of points inside it ($2^1$).
- In a 2D space (a flat sheet), doubling the radius of a circle quadruples the number of points ($2^2$).
- In a $d$-dimensional space, the volume of a hypersphere—and thus the number of points $N(r)$ captured within a radius $r$—scales exponentially with the dimension: $N(r) \propto r^d$.
As noted by Brand in Charting a Manifold, we can exploit this relationship by analyzing the growth rate of the number of points within a neighborhood relative to its radius. By identifying the scale where this growth rate peaks, we can isolate the true dimensionality of the underlying manifold from small-scale noise and large-scale curvature.
Bypassing Density Bias with Nearest Neighbors
While the volume-scaling intuition is elegant, real-world data is rarely distributed uniformly. It clusters into dense pockets and sparse voids. If we simply count points within a fixed radius, variations in local density will corrupt our dimension estimate.
To bypass this issue, modern techniques shift their focus from counting points within a fixed radius to measuring the distances to a point's nearest neighbors.
Levina et al. formalized this in Maximum Likelihood Estimation of Intrinsic Dimension by modeling the arrival of near neighbors as a spatial Poisson process. By applying maximum likelihood principles to these neighbor distances, they showed that the local dimension can be calculated as the inverse of the average log-ratio of the distances to a point's $k$ nearest neighbors.
An even more minimal manifestation of this principle is the TWO-NN estimator. As detailed by the authors of Estimating the intrinsic dimension of datasets by a minimal neighborhood information and leveraged by Facco et al. in The intrinsic dimension of protein sequence evolution, we can estimate the intrinsic dimension using only the distances to a point's first and second nearest neighbors ($r_1$ and $r_2$).
$$\frac{r_2}{r_1}$$
Crucially, on a locally flat manifold, the ratio of these two distances is entirely independent of the local data density. Because the density parameter cancels out mathematically, the distribution of these distance ratios depends solely on the dimension $d$. This allows for a clean, non-parametric estimation of dimensionality that remains completely unbiased by non-uniform sampling or complex clustering.
Local Intrinsic Dimensionality
Manifolds are rarely uniform; they twist, intersect, and vary in complexity from one region to another. A single global dimension estimate can easily obscure these nuances. Local Intrinsic Dimensionality (LID) refines this approach by calculating dimensionality metrics within the immediate neighborhood of a specific data sample.
This localized lens has proven to be an invaluable diagnostic tool for understanding the inner workings and vulnerabilities of deep neural networks.
Tracking Representation Dynamics
Neural networks do not process information with a static geometry. Tan et al. demonstrated in Low-dimensional intrinsic dimension reveals a phase transition in gradient-based learning of deep neural networks that networks undergo distinct phase transitions between representation expansion and compression over the course of training.
This geometric evolution is also visible during a single forward pass. Joshi identified a consistent "hunchback" pattern in Large Language Models in Geometry of Decision Making in Language Models: internal token representations initially expand in intrinsic dimensionality to digest the contextual complexity of a prompt, and then sharply compress as the layers converge on a final, low-dimensional decision.
Detecting Adversarial Attacks and Anomalies
LID also exposes the structural fragility of high-dimensional spaces. Furon in On the Vulnerability of Retrieval in High Intrinsic Dimensionality Neighborhood and Bailey 0001 et al. in High Intrinsic Dimensionality Facilitates Adversarial Attack: Theoretical Evidence proved that neighborhoods with high intrinsic dimensionality become intensely crowded. In these dense regions, the distance between points effectively homogenizes, meaning an attacker needs only a microscopic perturbation to radically alter nearest-neighbor rankings.
Because adversarial manipulations and malicious inputs force data points off the clean data manifold into these high-dimensional, crowded spaces, they leave a distinct geometric signature. Defense frameworks, such as those proposed by Dras et al. in Detecting Textual Adversarial Examples Based on Distributional Characteristics of Data Representations and Jin et al. in A Unified Framework for Analyzing and Detecting Malicious Examples of DNN Models, use LID scoring across network layers to efficiently flag and filter out malicious or out-of-distribution inputs.
Diagnosing Model Memorization
In generative modeling, LID helps resolve long-standing conceptual paradoxes. Kamkari et al. utilized LID in A Geometric Explanation of the Likelihood OOD Detection Paradox to explain why deep generative models assign deceptively high likelihoods to simple out-of-distribution noise: these anomalous regions exhibit high local density but possess a significantly lower intrinsic dimensionality than the actual training data.
Furthermore, Ross et al. showed in A Geometric Framework for Understanding Memorization in Generative Models that when a generative model overfits and memorizes training samples, the local output space collapses into low-dimensional replicas of those training points. By measuring the local intrinsic dimension of generated outputs, practitioners can identify regions of manifold collapse and detect privacy-leaking generations without needing direct access to the original training set.
Key Foundational Work
Charting a Manifold (2002) — contributed 2 concept elements to this direction.