TL;DR
This paper reveals a critical flaw in how we measure the "intrinsic dimension" (ID) of neural network representations: common estimators don't actually track the true underlying dimension. Instead, they capture a phenomenon related to how representations spread out in latent space, which is more akin to entropy. This finding challenges interpretations of previous ID studies and suggests a new way to understand representation geometry.
What Problem Does This Paper Address?
Deep learning models are incredibly complex, operating in high-dimensional spaces. To understand them better, researchers often look at the "intrinsic dimension" (ID) of their internal representations. The idea is that even though these representations live in a high-dimensional space (like the pixels of an image), they might actually lie on or near a much lower-dimensional "manifold." This manifold hypothesis suggests that the "true" dimensionality of the data is much smaller than the ambient space.
However, previous studies have observed a consistent pattern: as data flows through a neural network, the estimated ID tends to increase in early layers and decrease in later layers. This has been interpreted as the emergence of abstractions or phase transitions. This paper questions whether these observed ID patterns truly reflect the underlying manifold's dimension or if there's a disconnect between theory and practice.
Methods
The authors tackle this problem through a combination of theoretical analysis and empirical investigation.
Theoretical Contributions: * They prove that for Lipschitz continuous neural network layers (which most standard layers are), the intrinsic dimension (both Hausdorff and pointwise) cannot increase as data passes through the network. This is a fundamental theoretical constraint. * They extend this to the "union of manifolds" hypothesis, showing the same non-increasing property holds. * They demonstrate that for Large Language Models (LLMs), due to the discrete nature of tokens, the intrinsic dimension of their embeddings and representations is theoretically zero.
Empirical Investigations: * They analyze layer-wise ID estimates from various vision models (CNNs, ViTs) and LLMs. * They investigate the underlying factors driving the observed ID patterns, focusing on: * NN distances: How distances between nearest neighbors change across layers. * Cosine Similarity: How similar representations are to each other. * Representation Length (L2 norm): How far representations are from the origin. * Entropy: A measure of how spread out the variance of representations is.
Datasets
The paper uses several standard datasets for their experiments:
- ImageNet: For training and evaluating convolutional neural networks (CNNs) and vision transformers (ViTs).
- Wikitext: For training and evaluating large language models (LLMs).
They also generate synthetic datasets (e.g., points sampled from a dm-hyperball) to control the true intrinsic dimension and study estimator behavior.
Key Insights and Ideas
- ID Estimators are Misleading: The core insight is that common ID estimators (like TwoNN and MLE) do not track the true intrinsic dimension of neural representations. They are heavily biased, especially in high dimensions, and their layer-wise patterns are not indicative of the underlying manifold dimension.
- The "ID Increase" is an Artifact: The observed increase in estimated ID in early layers is not due to the emergence of higher-dimensional abstractions. Instead, it's driven by the increasing separation of representations in latent space, which causes NN distances to grow.
- Entropy is the Real Story: The paper strongly suggests that the observed layer-wise ID patterns are actually capturing changes in the entropy of the representation distribution. As representations spread out in latent space (increasing entropy), ID estimators tend to increase, and vice-versa. This provides a more theoretically grounded interpretation.
- LLM Embeddings Have Zero ID: A surprising theoretical result is that LLM token embeddings, due to the discrete nature of tokens, have a strict topological intrinsic dimension of zero. This is fundamentally different from image representations.
Key Results
- Theoretical Proofs: The paper provides mathematical proofs showing that intrinsic dimension cannot increase across Lipschitz layers in neural networks.
- Empirical Evidence: Experiments across various LLMs, CNNs, and ViTs consistently show that estimated layer-wise IDs increase in early layers, contradicting the theoretical findings.
- Bias in Estimators: Figures demonstrate a significant negative bias in common ID estimators, which grows with the true intrinsic dimension.
- Correlation with Entropy: Layer-wise ID patterns strongly correlate with von Neumann entropy patterns across different model types, suggesting entropy is the underlying phenomenon being captured.
- LLM Specifics: LLM embeddings and representations are shown to have a theoretical ID of zero.
Limitations and Future Work
Limitations: * The theoretical results on ID non-increase apply to Lipschitz layers. While most standard layers are Lipschitz, some operations (like hard quantization or certain types of self-attention) might violate this, though the authors discuss how to handle these in compact domains. * The paper focuses on pointwise and Hausdorff dimensions. Other notions of intrinsic dimension exist.
Future Work: * Develop Better Estimators: The authors suggest designing new ID estimators that are more reliable and adhere to theoretical constraints (e.g., non-increasing layer-wise IDs). * Formalize Entropy-ID Connection: Further investigation into the theoretical connection between ID and entropy is proposed as a promising direction. * Broader Implications: Re-evaluating conclusions drawn from previous ID estimation studies across various domains (vision, NLP) is recommended. * Understanding Representation Expansion: Further exploring why representations expand in latent space and how this expansion is structured (e.g., directionally) is suggested.
Summary updated