Understanding Undesirable Word Embedding Associations

Kawin Ethayarajh, David Duvenaud, Graeme Hirst

TL;DR

This paper provides a rigorous mathematical foundation for understanding how word embeddings capture and amplify social biases. The authors prove that post-hoc subspace projection can perfectly debias embeddings, reveal critical mathematical flaws in the widely used Word Embedding Association Test (WEAT), and introduce a robust new metric called Relational Inner Product Association (RIPA) to accurately track how embedding algorithms systematically amplify data stereotypes.

What Problem Does This Paper Address?

Word embeddings (like SGNS and GloVe) are notorious for picking up undesirable human biases from text corpora—famously reflecting stereotypes like doctor:nurse :: man:woman. While the community has developed tools to measure these biases (such as WEAT) and mitigate them (such as subspace projection), we haven't fully understood why or how well these methods work under the hood.

This paper tackles three major open questions:

  1. What theoretical guarantees do we actually have that post-hoc debiasing methods work?
  2. Is WEAT truly a reliable metric for measuring vector space associations?
  3. Does the embedding model itself generate new bias, or does it simply mirror the training data?

Methods

The authors approach these problems by viewing word embedding algorithms through the lens of implicit matrix factorization:

  • Theoretical Proof of Debiasing: They mathematically prove that for any model implicitly factorizing a word-context matrix, debiasing vectors post-hoc via subspace projection is equivalent to training on a perfectly unbiased corpus. Crucially, they show that vectors should not be normalized before debiasing, and the bias subspace must be defined by the span of the attribute vectors rather than just a principal component.
  • Deconstructing WEAT: They analyze the math behind the Word Embedding Association Test (WEAT) and demonstrate that its reliance on cosine similarity introduces a systematic flaw: it implicitly requires attribute words to appear with equal frequency in the training corpus.
  • Relational Inner Product Association (RIPA): To replace WEAT, they derive RIPA: $\beta(\vec{w};\vec{b}) = \langle \vec{w},\vec{b} \rangle$, where $\vec{b}$ is the first principal component of a set of target difference vectors. RIPA avoids cosine normalization, making it directly interpretable in terms of pointwise mutual information (PMI).
  • Unsupervised Debiasing Selection: Previous methods used supervised learning (SVMs) to separate words that should be gendered (like "queen") from those that shouldn't (like "nurse"). The authors introduce a simpler, unsupervised heuristic based on RIPA thresholds to protect gender-appropriate words during debiasing.

Datasets

The authors evaluate their insights using Skip-gram with Negative Sampling (SGNS) embeddings trained on a Wikipedia corpus. They utilize word lists from Bolukbasi et al. (2016) split into three categories:

  • Gender-Appropriate: 164 words that are gender-specific by definition (e.g., queen, king, mom, dad).
  • Gender-Biased: 68 words reflecting social stereotypes (e.g., nurse, doctor, housekeeper, architect).
  • Gender-Neutral: A control group of 200 randomly sampled words that appear at least 10,000 times in the corpus.

Key Insights and Ideas

  • WEAT is Easily Gamed: Because WEAT ignores word frequency discrepancies, it can be easily manipulated. By carefully selecting attribute sets, virtually any completely neutral word (like "door" or "dog") can be classified as highly male- or female-biased with strong statistical significance.
  • Vector Length Carries Crucial Information: Prior work recommended normalizing word vectors to a unit length before debiasing. This paper reveals that doing so destroys frequency information, breaking the mathematical guarantees of the debiasing process.
  • The Amplification Effect: On average, SGNS does not make neutral words any more gendered than they are in the raw text. However, for words that already possess some gender association (whether stereotypical or definitional), the embedding process amplifies that association. This happens because of second-order similarity: words with similar contextual profiles get pushed closer together in a low-dimensional space than their raw co-occurrence frequencies would justify.

Key Results

  • WEAT Vulnerability: The authors show how the target words \{door\} vs. \{curtain\} can yield a maximal WEAT effect size of $2.0$ (purely male-associated) or $-2.0$ (purely female-associated) simply by swapping out the attribute word sets (e.g., using masculine/feminine vs. woman/man).
Extracted table showing how WEAT can be manipulated
  • Quantifying Amplification: Using RIPA, the authors found that the absolute genderedness of neutral words changed by a negligible $-0.006$ from corpus to embedding space. However, gender-biased words saw an absolute increase of $0.197$, and gender-appropriate words jumped by $0.291$.
Extracted table showing genderedness changes from corpus to embedding space
  • Superior Unsupervised Debiasing: The authors' unsupervised selection method successfully preserved 94.9% of gender-appropriate analogies while filtering out biased ones. The previous supervised SVM approach preserved only 16.5% of appropriate analogies while erroneously leaving 80.0% of biased ones intact.
Analogy preservation curves comparing the unsupervised method to Bolukbasi et al.

Limitations and Future Work

The primary limitation is that the debiasing theorem assumes perfect reconstruction (i.e., zero reconstruction error during matrix factorization), which is never fully achieved in practical, lower-dimensional embeddings. Additionally, while the mathematical proofs are generalizable to any binary association, the empirical experiments in this paper were strictly limited to gender bias. The authors leave the exploration of other bias types (e.g., race, religion) to future work.

Summary updated

Abstract

Word embeddings are often criticized for capturing undesirable word associations such as gender stereotypes. However, methods for measuring and removing such biases remain poorly understood. We show that for any embedding model that implicitly does matrix factorization, debiasing vectors post hoc using subspace projection (Bolukbasi et al., 2016) is, under certain conditions, equivalent to training on an unbiased corpus. We also prove that WEAT, the most common association test for word embeddings, systematically overestimates bias. Given that the subspace projection method is provably effective, we use it to derive a new measure of association called the $\textit{relational inner product association}$ (RIPA). Experiments with RIPA reveal that, on average, skipgram with negative sampling (SGNS) does not make most words any more gendered than they are in the training corpus. However, for gender-stereotyped words, SGNS actually amplifies the gender association in the corpus.
BibTeX
@inproceedings{ethayarajh2019understanding,
  author = {Kawin Ethayarajh and David Duvenaud and Graeme Hirst},
  title = {{Understanding Undesirable Word Embedding Associations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=yT7ZN1wO6vi},
}

Loading related content...

Figure