A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations

Zekun Yang, Juan FengAAAI 2020 · DOI

Introduction

Word embeddings like Word2Vec and GloVe are fundamental to modern Natural Language Processing (NLP), yet they notoriously inherit societal biases from their training data. A common example is the vector for "doctor" being significantly closer to "man" than to "woman." While previous debiasing efforts have attempted to address this, they often fail to remove deeper, relational biases that persist even after geometric adjustments.

The Problem of Relational Bias

Most existing debiasing techniques rely on orthogonal projection. These methods identify a "gender direction" (e.g., the vector difference between he and she) and flatten other word vectors so they have zero projection onto that axis. However, research has shown that this approach merely hides the bias. Biased words like "nurse" and "receptionist" continue to cluster together because they share a latent, relational bias that simple geometric projection cannot reach. This paper proposes a robust causal framework to solve this "relational bias" problem.

Methodology: Half-Sibling Regression (HSR)

The authors introduce Half-Sibling Regression (HSR), a post-processing algorithm based on the principle of confounding-noise elimination. The core innovation is treating gender-definition word vectors ($V_D$, such as mother/father) and biased non-gender-definition vectors ($V_N$, such as nurse/doctor) as "half-siblings."

Figure 1: The causal graph of the HSR framework, illustrating how Gender Information acts as a confounding variable for observable word vectors.

As shown in the causal graph above, both types of vectors share a common "parent"—Gender Information—but possess different unique Semantic Information. The HSR process follows two main steps:

  1. Predict the Bias: Using Ridge Regression, the model learns to predict the information in $V_N$ using $V_D$. Because $V_D$ contains almost no semantic information other than gender, the resulting prediction ($\hat{G}$) isolates the spurious gender noise.
  2. Subtract the Noise: The final debiased vector is calculated by subtracting this predicted noise from the original vector: $\hat{V}_N = V_N - \hat{G}$.

This method is computationally efficient, requiring only one hyperparameter (the Ridge Regression constant $\alpha$) without needing to retrain the entire embedding model.

Key Results and Evaluation

The authors evaluated HSR using GloVe embeddings across several benchmarks, including the GBWR (Gender-Biased Word Relation) suite and the WinoBias dataset for coreference resolution.

Figure 4: Visualization of word embeddings after HSR-GloVe processing, showing the distribution of professional terms.

Relational Bias Reduction

HSR achieved a 6.72% and 4.98% improvement over the previous state-of-the-art (Hard-GloVe) on relational bias tasks. By subtracting the predicted gender component, HSR breaks the hidden clusters of biased words that previous methods left untouched.

Semantic Enhancement as Denoising

One of the most striking findings is that debiasing actually improved performance on standard word similarity tasks by nearly 7%, and sentence-level tasks by 14.19%. This suggests that gender bias acts as a form of statistical noise; removing it allows the true semantic relationships between words to emerge more clearly.

Downstream Fairness

On the WinoBias task, HSR reduced the performance gap between pro-stereotypical and anti-stereotypical sentences by 87.26% compared to the runner-up method. This indicates that the model becomes significantly fairer in practical applications like coreference resolution.

Conclusion

HSR represents a shift from geometric to causal debiasing. By treating bias as a confounding variable to be regressed out, the method effectively reduces relational bias while simultaneously improving the overall semantic quality of word embeddings. Future work will focus on extending this framework to contextual embeddings like BERT and GPT and exploring its efficacy against other forms of social bias.

Summary updated

BibTeX
@inproceedings{yang2020causal,
  author = {Zekun Yang and Juan Feng},
  title = {{A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6486},
}
Figure