The Illusion of Relational Arithmetic in Word Embeddings

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 27 papers20 conceptsfrozen 2026-10-04

The idea that word embeddings like Word2Vec or GloVe can perform "linguistic algebra" is one of the most famous results in modern NLP. The classic example—$\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$—suggests that these high-dimensional spaces capture abstract semantic relations as linear offsets. However, a growing body of research suggests that this "magic trick" is less about deep relational reasoning and more about the simple geometric properties of local neighborhoods.

The Proximity Trap

The core intuition behind vector analogies is that the offset between two words (like man and woman) represents a "gender direction" that can be applied elsewhere. But in The (too Many) Problems of Analogical Reasoning with Word Vectors, Anna Rogers, Aleksandr Drozd, and Bofang Li argue that this success is often a byproduct of neighborhood density.

When we perform the calculation $A - B + C = D$, we expect the resulting vector to point toward a distant, semantically appropriate $D$. In reality, the vector $A - B$ is often so small that the result of the calculation remains physically closest to the starting word $C$. The "reasoning" isn't moving across the space; it's barely moving at all. If $D$ happens to be a near neighbor of $C$ to begin with (as queen is to woman), the model appears to have "solved" the analogy, even if the linear offset itself was mathematically negligible.

Hidden Evaluation Hacks

This illusion is maintained by specific choices in how we evaluate these models. In The Word Analogy Testing Caveat, Natalie Schluter points out a critical "hack" in standard benchmarks: the model is typically forbidden from predicting any of the input words ($A, B,$ or $C$) as the answer.

When this constraint is removed, the performance of vector arithmetic collapses. The model frequently outputs $C$ as the most similar vector to $A - B + C$. This confirms that the vector difference between $A$ and $B$ is often not a meaningful "relational direction" but rather a source of noise that the model ignores by staying as close to $C$ as possible. The perceived success of the analogy is thus an artifact of the evaluation protocol rather than a fundamental property of the embedding space.

Bias as a Mathematical Artifact

The fragility of these offsets has significant implications for how we measure social bias in AI. A common technique for identifying gender bias is to look for analogies like $\vec{man} : \vec{computer\_programmer} :: \vec{woman} : \vec{homemaker}$.

However, Alison Sneyd, Mark Stevenson, and Haiyang Zhang demonstrate in Robustness and Reliability of Gender Bias Assessment in Word Embeddings that these results are highly unstable. They argue that such "biased" analogies are often mathematical artifacts of high-dimensional similarity. Because the formula used to solve analogies inherently captures general word similarity, a model might pair woman with homemaker simply because they appear in similar contexts, not because the model has encoded a specific linear "stereotype" direction.

Why It Matters

If relational reasoning in embeddings is an illusion of local density, then using vector offsets to "fix" models—such as by subtracting a "gender dimension"—may be ineffective or even misleading. Understanding that these models operate on local spatial proximity rather than abstract logical directions forces a move away from simple linear arithmetic toward more robust ways of representing and de-biasing semantic relationships.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 6 papers
  2. Version 1 · 2026-10-04 · 7 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spatial_similarity_bias_2c236253,
  title = {spatial similarity bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-similarity-bias/rdr_2c236253869f486cbb4431af846df561},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_illusion_of_relational_arithmetic_in_2c236253_v1,
  title = {The Illusion of Relational Arithmetic in Word Embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-illusion-of-relational-arithmetic-in-word-embeddings/rdr_2c236253869f486cbb4431af846df561/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}