Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space

Kawin Ethayarajh

Abstract

A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, $\textit{gender}$ corresponds to $\vec{\textit{woman}} - \vec{\textit{man}}$ and $\vec{\textit{queen}} - \vec{\textit{king}}$. This, in turn, allows word analogies to be solved arithmetically: $\vec{\textit{king}} - \vec{\textit{man}} + \vec{\textit{woman}} \approx \vec{\textit{queen}}$. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models $\textit{exclusively}$ represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for $\textit{gender}$, we can find an orthogonal matrix $R$, representing a rotation and reflection, such that $R(\vec{\textit{king}}) \approx \vec{\textit{queen}}$ and $R(\vec{\textit{man}}) \approx \vec{\textit{woman}}$. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.

Introduction

Since the rise of Word2Vec and GloVe, the NLP community has largely accepted the "linear substructure" hypothesis: that semantic and syntactic relationships (like gender or verb tense) are encoded as constant translation vectors. However, mathematical proofs have shown that the conditions required for these analogies to hold perfectly are rarely met in real-world training corpora.

This research, conducted by Kawin Ethayarajh, asks a fundamental question: Are translations the only simple way to represent word relationships? By moving beyond simple vector arithmetic to explore orthogonal and general linear transformations, we can identify more flexible yet parsimonious frameworks for how semantic relationships are processed in embedding spaces. This is crucial because if relationships are more than just translations, current methods for evaluating embedding quality and "debiasing" models might be fundamentally incomplete.

Methodology

The paper compares three ways to transform a source word vector ($\vec{x}$) into a target word vector ($\vec{y}$):

  1. Translative: The classic approach where $\vec{y} \approx \vec{x} + \vec{b}$. The vector $\vec{b}$ is calculated as the mean difference between all known pairs in a category.
  2. Orthogonal: A transformation $\vec{y} \approx R\vec{x}$ where $R$ is an orthogonal matrix (representing rotation and reflection). This is solved using the Orthogonal Procrustes solution.
  3. Linear: The most general form, $\vec{y} \approx A\vec{x}$, where $A$ is any square matrix, solved using Ordinary Least Squares (OLS).

To ensure the transformations capture global relationships rather than overfitting to specific pairs, the author uses the translation vector $\vec{b}$ to "seed" the learning of the matrices. A large set of random word vectors $X$ is shifted by $\vec{b}$ to create $Y$, and the matrix is then learned to map $X$ to $Y$.

Key Results

Performance and Accuracy

On the standard Mikolov et al. analogy task, general linear transformations achieved the highest average accuracy (0.798), outperforming the classic translative approach (0.782). Orthogonal transformations, despite the rigid constraint of preserving inner products, were almost as accurate (0.761) as vector arithmetic. This proves that "rotating" a vector is a mathematically valid alternative to "adding" a relationship vector.

Comparison of accuracy and cosine similarity across analogy categories for different transformation methods.

Data Efficiency

The transformations are remarkably robust and data-efficient. As shown in the performance curves, accuracy plateaus after sampling only 250–500 words to learn the transformation matrix. This suggests that these linear relationships are deeply embedded in the global geometry of the word embedding space.

Average accuracy as a function of the number of sampled words used to learn the transformation.

Discussion

Transformer Intuition

The paper suggests that a single attention head in a Transformer model has exactly the right mathematical capacity to represent these linear and orthogonal relationships. This offers a bridge between low-level embedding properties and high-level model architecture, suggesting that Transformers may naturally process semantic relationships through these transformations.

Implications for Debiasing

Most current "debiasing" techniques work by collapsing a single "bias direction." If bias (such as gender) is actually better represented as an orthogonal transformation (a rotation), then simply shifting vectors along a line will not truly remove the bias. These findings suggest a need for new debiasing strategies based on rotational geometry.

Conclusion

Word relationships are not limited to simple additions. Linear and orthogonal maps sit in a "sweet spot" of complexity that effectively represents semantic structures. Linear transformations, in particular, provide a superior representation to traditional translations, offering a more robust framework for understanding and manipulating word embedding spaces.

Summary updated

Abstract

A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, $\textit{gender}$ corresponds to $\vec{\textit{woman}} - \vec{\textit{man}}$ and $\vec{\textit{queen}} - \vec{\textit{king}}$. This, in turn, allows word analogies to be solved arithmetically: $\vec{\textit{king}} - \vec{\textit{man}} + \vec{\textit{woman}} \approx \vec{\textit{queen}}$. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models $\textit{exclusively}$ represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for $\textit{gender}$, we can find an orthogonal matrix $R$, representing a rotation and reflection, such that $R(\vec{\textit{king}}) \approx \vec{\textit{queen}}$ and $R(\vec{\textit{man}}) \approx \vec{\textit{woman}}$. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.
BibTeX
@inproceedings{ethayarajh2019rotate,
  author = {Kawin Ethayarajh},
  title = {{Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=0sY6FWkHvIt},
}

Loading related content...

Figure