The classic vector arithmetic equation $\text{king} - \text{man} + \text{woman} \approx \text{queen}$ is one of the most famous results in natural language processing. For years, it served as elegant proof that word embeddings capture structured, human-like semantic relationships through parallel geometric offsets.
However, a deeper look into the geometry of high-dimensional spaces reveals that this "vector magic" is largely an illusion. Instead of executing deep relational logic, standard analogy evaluations frequently succeed due to a simpler, confounding geometric property: spatial proximity and local neighborhood density.
Why the Math Tricks Us
To understand why the analogy test misleads us, we must look at how the target word is selected. In the standard offset method, we take three vectors ($a$, $b$, and $c$) and calculate a query vector:
$$\vec{v}_{\text{query}} = \vec{v}_c + (\vec{v}_a - \vec{v}_b)$$
We then search the embedding space for a word $d$ whose vector maximizes cosine similarity to $\vec{v}_{\text{query}}$.
As Linzen demonstrated in Issues in evaluating semantic spaces using word analogies, this method fundamentally conflates true semantic offset consistency with simple neighborhood similarity. If the target word $d$ is already an extremely close spatial neighbor of $c$ or $a$ to begin with, the offset vector $(\vec{v}_a - \vec{v}_b)$ does not actually need to perform any meaningful "travel" across the space to hit the target.
In Analogies minus analogy test: measuring regularities in word embeddings, Fournier et al. decomposed the analogy test to show that successful completions are heavily dominated by this initial proximity. Because target words are typically already top spatial neighbors of the input words, the mathematical representation of the underlying relationship can be weak or non-existent, yet the model will still pass the test. The arithmetic operation contributes remarkably little to the final prediction; it merely hovers within the same dense local neighborhood.
The Banned-Input Artifact
If the vector arithmetic doesn't actually move the query vector very far, why does it return the correct answer so often? The secret lies in a subtle flaw in the evaluation protocol itself.
In The (too Many) Problems of Analogical Reasoning with Word Vectors, Rogers et al. pointed out that standard evaluation protocols artificially inflate accuracy by explicitly excluding the input words ($a$, $b$, and $c$) from the pool of allowed answers.
When you calculate $\vec{v}_{\text{king}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}}$, the resulting vector actually remains closest to the original starting points—king and woman. Because the evaluation protocol forbids the system from guessing the input words, it is forced to select the next closest word in that immediate neighborhood, which happens to be queen. When Rogers et al. allowed these inputs to be valid answers, the linear arithmetic frequently failed to bridge the semantic gap, remaining stuck at the starting point and exposing the lack of true relational reasoning.
Standard Analogy Search:
[Query Vector] ──> (Closest: "woman" [BANNED]) ──> (Next Closest: "queen" [CORRECT])
Confounding Bias and Representation
This spatial proximity confounding has serious implications for how we interpret what word embeddings learn. Historically, representing words via co-occurrence patterns has struggled to capture nuanced, abstract relationships without being obscured by local clustering, a limitation explored early on by Ma et al. in Self-Organizing Chinese and Japanese Semantic Maps. The analogy test was supposed to prove that modern embeddings had overcome this, but the proximity bias suggests otherwise.
Worse, this geometric artifact can distort downstream sociological conclusions. For instance, the famous analogy "man is to computer programmer as woman is to homemaker" has been widely cited as evidence of human-like societal stereotypes embedded in vector spaces.
However, Sneyd et al. demonstrated in Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs that these biased-looking analogies are frequently mathematical artifacts of high-dimensional vector space rather than a direct encoding of societal stereotypes. Because the underlying formulas capture general word similarity and local neighborhood density, pairs that happen to be close in context generate these analogies purely due to geometric proximity, making analogy-based bias detection highly fragile and sensitive to the choice of base words.
Disentangling Proximity from Relations
To truly determine whether a model captures semantic transformations, researchers must separate directional regularities from local density. Prominent theoretical attempts to defend the analogy phenomenon—such as the idea that word pairs act as distributional paraphrases of one another—have been empirically debunked by Fournier et al. in Paraphrases do not explain word analogies, proving that real-world corpus data does not support the paraphrase hypothesis.
Instead, moving forward requires metrics designed to bypass the proximity illusion entirely. In Analogies minus analogy test, the authors proposed alternative evaluation frameworks like Offset Concentration and Pairing Consistency. These metrics isolate whether parallel vector structures genuinely exist across different word pairs, ensuring that we measure true relational regularities rather than the accidental alignment of dense spatial neighborhoods.
Key Foundational Work
Issues in evaluating semantic spaces using word analogies (2016) — contributed 5 concept elements to this direction.
Self-Organizing Chinese and Japanese Semantic Maps (2002) — contributed 1 concept elements to this direction.