Modern deep learning operates in spaces of staggering dimensionality. A single ImageNet picture exists in a 150,528-dimensional space of raw pixels, and a Large Language Model might process embeddings in 4,096 dimensions. Yet, we know intuitively that "natural" data is not random noise; it possesses structure. The Manifold Hypothesis suggests that this high-dimensional data actually lies on or near a much lower-dimensional "surface" or manifold.
Local Intrinsic Dimensionality (LID) is a geometric tool used to measure the complexity of these manifolds. Unlike global dimensionality metrics, LID looks at the "effective" degrees of freedom in the immediate neighborhood of a specific data point.
The Intuition: Expansion and Volume
To understand LID, imagine placing a small ball around a data point and gradually increasing its radius.
- In a 1D line, doubling the radius doubles the number of neighbors you find.
- In a 2D plane, doubling the radius quadruples the number of neighbors (area $\propto r^2$).
- In an $m$-dimensional space, the number of neighbors grows at a rate proportional to $r^m$.
LID formalizes this by measuring the rate at which the cumulative number of neighbors grows as the distance from a reference point increases. As Xingjun Ma and Sarah Monazam Erfani describe in their foundational work, LID identifies whether a sample resides on a low-dimensional manifold or has been pushed into the "ambient" high-dimensional space.
Measuring the Unmeasurable
How do we calculate this without knowing the underlying manifold? Researchers primarily use two statistical estimators:
- Maximum Likelihood Estimation (MLE): Proposed by Elizaveta Levina and Peter J. Bickel, this method models the distribution of distances to a point’s $k$-nearest neighbors as a Poisson process. The LID is the inverse of the average log-ratio of these distances.
- The Distance Ratio (twoNN): A more computationally efficient approach used by Alessio Ansuini and Alessandro Laio. It estimates dimensionality by looking only at the ratio of distances between a point and its first and second nearest neighbors.
More recently, Hamidreza Kamkari has shown that diffusion models can estimate LID by analyzing how probability density evolves as noise is added to the data.
Why Adversarial Examples "Pop Out"
One of the most striking applications of LID is in AI safety. Xingjun Ma and colleagues discovered that adversarial perturbations—tiny changes that fool a model—work by pushing data points off the natural manifold.
While an adversarial image might look identical to a clean one, its representation in the hidden layers of a neural network resides in a much higher-dimensional region. It "pops out" of the thin manifold of natural data into the vast, empty ambient space. By measuring the LID of a sample's activations, we can detect these attacks with high accuracy because their "space-filling" properties are fundamentally different from benign data. This same logic applies to detecting "jailbreak" prompts in large language models, as shown by Canaan Yung.
The Geometry of Learning and Generalization
LID also provides a window into how neural networks learn. Anthony L. Caterini and Gabriel Loaiza-Ganem found that better generalization correlates with lower intrinsic dimension in the final layers. Essentially, a model that generalizes well has learned to compress complex inputs into a simple, low-dimensional conceptual space.
This compression isn't uniform. In Transformers, Lucrezia Valeriani and Alessandro Laio observed a bell-shaped curve of dimensionality:
- Initial Layers: The model expands the data to resolve features.
- Middle Layers: The "sweet spot" where semantic information is most richly organized.
- Final Layers: The model compresses the representation to make a specific prediction.
Beyond Detection: Architecture and Pruning
Understanding the local geometry of data allows for more than just defense. Jiangchao Yao used LID to speed up Neural Architecture Search, grouping similar models based on their representational geometry rather than expensive gradients. Similarly, Hanzhang Wang demonstrated that Vision-Language Models can be pruned more effectively by identifying layers with low intrinsic dimensionality—essentially removing the "simple" or redundant parts of the network while preserving the complex, high-ID visual features.
By treating neural networks as geometric engines that fold and unfold manifolds, LID offers a precise language to describe why models fail, how they generalize, and how we can make them more efficient.