When we train deep language models, we map words, sentences, or images into high-dimensional vector spaces—typically consisting of hundreds or thousands of dimensions. Mathematically, a high-dimensional space is unimaginably vast, offering an exponential number of orthogonal directions for representations to spread out.
Yet, if you extract raw embeddings from pre-trained models like BERT or CLIP and measure their distribution, you will find something deeply counterintuitive: instead of utilizing the full expanse of the latent space, the vectors huddle together in a narrow, highly directional cone. This phenomenon is known as embedding anisotropy or representation degeneration.
Understanding why this geometric collapse happens, how it warps our evaluation metrics, and how to fix it is essential for building effective retrieval, alignment, and representation systems.
The Intuition of the Narrow Cone
To build an intuition for anisotropy, imagine a three-dimensional space where every vector points almost entirely along the positive z-axis. Even if two vectors are conceptually unrelated, they are forced to share a nearly identical directional orientation because the space itself is stretched out like a long, thin needle.
In an ideal, isotropic embedding space, vectors are uniformly distributed in all directions. This uniform coverage maximizes the expressiveness of the space, ensuring that distinct concepts point in distinct directions.
When a space becomes anisotropic, its effective dimensionality plummets. As Reimers et al. observed in the context of dense information retrieval, this spatial compression forces unrelated items into the same crowded neighborhood. When millions of documents are packed into a narrow subspace, the risk of vector collisions and retrieval errors increases dramatically at scale.
Why Does the Space Collapse?
Anisotropy is not a random defect; it is an emergent property of how transformers process token distributions and optimize their loss functions. Two primary mechanisms drive this collapse:
1. The Tyranny of Outlier Dimensions
The primary culprit behind the narrow cone is the emergence of a tiny handful of high-magnitude "rogue" or "outlier" dimensions. Kovaleva et al. demonstrated that a fraction of a percent of the dimensions in a transformer's hidden states dominate the variance of the entire space. These outlier dimensions act like a powerful gravitational pull, dragging every single vector toward a specific directional axis.
Interestingly, Timkey et al. showed that this anisotropy is often a local geometric artifact rather than a total structural failure. If you standardize or remove these few rogue dimensions, a highly balanced and isotropic underlying semantic structure is revealed.
2. Word Frequency and Token Dominance
During pre-training, language models must learn to predict both rare and highly frequent tokens (like punctuation or stop words). To optimize the cross-entropy loss, the model pushes the embeddings of high-frequency tokens toward the center of the representation space. Because these frequent tokens dominate the gradients during training, they warp the geometry of the entire space, pulling the representations of rarer words along with them into a dense, biased cluster.
The Illusion of Cosine Similarity
The most immediate casualty of an anisotropic space is cosine similarity, the standard metric used to evaluate semantic closeness. Cosine similarity calculates the cosine of the angle between two vectors:
$$\text{Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$$
If all vectors are crammed into a tight cone, the angle between any two vectors is necessarily tiny. Consequently, their cosine similarity will always be close to 1.0.
As Li et al. pointed out, this creates a severe calibration problem. When unrelated sentences exhibit high baseline cosine similarity, a model struggles to distinguish fine-grained semantic differences from background geometric bias. The metric reflects the shape of the vector space rather than genuine semantic integration.
This crowding also breaks tasks that rely on precise vector arithmetic. For instance, computing proportional analogies (e.g., "king - man + woman = queen") requires a directionally diverse space. When the space lacks directional diversity, calculating these precise relational offsets becomes nearly impossible.
Reshaping the Space: Two Paths to Isotropy
Researchers have developed two primary strategies to counteract anisotropy: post-processing calibration and contrastive training.
Post-Processing: Normalizing Flows and Whitening
If a pre-trained model's space is already collapsed, we can apply a training-free geometric transformation to "spread out" the vectors.
- Normalizing Flows (BERT-flow): Li et al. introduced an unsupervised method that uses normalizing flows to mathematically map the anisotropic, cone-shaped embedding distribution into a smooth, isotropic Gaussian distribution.
- Whitening: Another elegant approach is statistical whitening. By computing the mean and covariance matrix of the embeddings, we can linearly transform the space to ensure that all dimensions have a variance of one and zero covariance between them. Betser et al. demonstrated that whitening the latent space of contrastive models like CLIP allows the space to behave like a clean probability distribution, enabling direct likelihood and anomaly calculations.
In-Training: Contrastive Regularization
Instead of fixing the space after training, we can design objectives that inherently resist collapse. Contrastive learning frameworks, such as SimCSE proposed by Gao et al., optimize two simultaneous forces: alignment (pulling semantically identical pairs together) and uniformity (pushing random negative pairs apart).
The uniformity pressure forces the model to utilize the entire hypersphere, naturally shattering the narrow cone and distributing representations evenly across the latent space.
Anisotropic Space (Collapsed) Isotropic Space (Uniform)
\ | / _.._
\ | / .' `.
\ | / / o \
\|/ |o o|
v | o |
(Narrow Cone) \ /
`.____.'
(Hypersphere)
Is Anisotropy Always Harmful?
While the prevailing consensus in NLP treats anisotropy as a defect that limits model expressiveness, the reality contains important nuances.
Ait Saada et al. challenged the assumption that anisotropy inherently degrades performance. Through extensive text clustering experiments, the authors demonstrated that while anisotropy warps absolute directional distances, it often preserves the underlying topological relationships between documents. In other words, even if all items are squeezed into a cone, the correct neighbors often remain closest to each other.
The harm of anisotropy is therefore highly task-dependent. For tasks that rely on absolute thresholding or dense, large-scale indexing, correcting the geometric bias is critical. For tasks that merely require relative ranking, the model's internal logic may remain perfectly intact despite the crowded geometry.