In the early days of word embeddings, the "geometry" of the space seemed magical: $king - man + woman \approx queen$. However, as we have moved toward massive embedding dimensions in models like BERT, GPT, and Llama, this geometry has become increasingly counter-intuitive. Recent research reveals that high-dimensional embedding spaces suffer from a "distance collapse" where traditional metrics lose their ability to distinguish between degrees of similarity.
The Phenomenon of Near-Orthogonality
The core problem stems from a geometric property of high-dimensional spaces: as dimensionality increases, randomly chosen vectors tend to become nearly orthogonal to one another. In the context of language models, this means that most word vectors are effectively "at right angles" to each other.
Ryoma Sato, Hisashi Kashima, and Makoto Yamada demonstrate in Re-evaluating Word Mover's Distance that this leads to a bimodal distance distribution. In these spaces, any two non-identical tokens are roughly equidistant. If you measure the distance between "cat" and "dog" versus "cat" and "refrigerator," the difference becomes negligible.
This loss of contrast makes it difficult for models to differentiate between words that are genuinely unrelated and those that are merely somewhat related, a problem highlighted by Behrouz Haji Soleimani and Stan Matwin in their work on KUBWE. When every point appears nearly equidistant, the "resolution" of the semantic space vanishes.
Why Complex Metrics Collapse
This geometric constraint has a surprising side effect: it renders sophisticated distance metrics redundant. Word Mover’s Distance (WMD) was designed to measure the "travel cost" between documents by looking at the semantic distances between their individual words. However, because high-dimensional word vectors are nearly orthogonal, the "cost" to move between any two different words is almost always the same.
As a result, WMD collapses into a behavior nearly identical to a simple, normalized Bag-of-Words (BOW) model. Sato et al. showed that the perceived superiority of WMD was often just an artifact of preprocessing; once BOW is properly L1-normalized, it performs just as well because the underlying embedding geometry doesn't actually provide the "semantic nuance" WMD was trying to exploit.
The Retrieval Gap
Even when vectors seem aligned, they may not be "near" each other in a way that is useful for search. Zhijin Guo and Martha Lewis found in Quantifying Compositionality of Classic and State-of-the-Art Embeddings that high cosine similarity does not necessarily translate to high retrieval accuracy.
In massive vector spaces, two vectors can have high directional alignment (pointing in the same general direction) without being each other's nearest neighbors. This "curse of dimensionality" makes simple $k$-nearest neighbor retrieval unreliable for tasks like selecting examples for in-context learning, as noted by Soumya Suvra Ghosal and colleagues in their work on KITE.
Restoring Semantic Contrast
To combat this collapse, researchers are moving away from direct distance measurements in favor of methods that respect the local manifold of the data:
- Kernel Methods: Algorithms like KITE and KUBWE use the "kernel trick" to project embeddings into spaces where semantic differences are more discriminative.
- Graph Diffusion: Dominik J. Schindler and Mauricio Barahona propose LGDE, which treats embeddings as a geometric network. By using graph diffusion, they can discover "insider" jargon and related words that standard similarity measures miss because they are buried in the uniform distance noise.
- Layer-Specific Analysis: Since semantic compositionality is dynamic—often peaking in the middle layers of a Transformer—Jian Gu and others suggest SALF, which tracks how "meaning" evolves layer-by-layer rather than treating the final output embedding as the sole source of truth.
By acknowledging that high-dimensional spaces are naturally "sparse" and "orthogonal," these researchers are building models that can see the semantic signal through the geometric noise.