Introduction
Word embeddings like Word2Vec and GloVe are fundamental to modern Natural Language Processing (NLP), yet they notoriously inherit societal biases from their training data. A common example is the vector for "doctor" being significantly closer to "man" than to "woman." While previous debiasing efforts have attempted to address this, they often fail to remove deeper, relational biases that persist even after geometric adjustments.
The Problem of Relational Bias
Most existing debiasing techniques rely on orthogonal projection. These methods identify a "gender direction" (e.g., the vector difference between he and she) and flatten other word vectors so they have zero projection onto that axis. However, research has shown that this approach merely hides the bias. Biased words like "nurse" and "receptionist" continue to cluster together because they share a latent, relational bias that simple geometric projection cannot reach. This paper proposes a robust causal framework to solve this "relational bias" problem.
Methodology: Half-Sibling Regression (HSR)
The authors introduce Half-Sibling Regression (HSR), a post-processing algorithm based on the principle of confounding-noise elimination. The core innovation is treating gender-definition word vectors ($V_D$, such as mother/father) and biased non-gender-definition vectors ($V_N$, such as nurse/doctor) as "half-siblings."
As shown in the causal graph above, both types of vectors share a common "parent"—Gender Information—but possess different unique Semantic Information. The HSR process follows two main steps:
- Predict the Bias: Using Ridge Regression, the model learns to predict the information in $V_N$ using $V_D$. Because $V_D$ contains almost no semantic information other than gender, the resulting prediction ($\hat{G}$) isolates the spurious gender noise.
- Subtract the Noise: The final debiased vector is calculated by subtracting this predicted noise from the original vector: $\hat{V}_N = V_N - \hat{G}$.
This method is computationally efficient, requiring only one hyperparameter (the Ridge Regression constant $\alpha$) without needing to retrain the entire embedding model.
Key Results and Evaluation
The authors evaluated HSR using GloVe embeddings across several benchmarks, including the GBWR (Gender-Biased Word Relation) suite and the WinoBias dataset for coreference resolution.
Relational Bias Reduction
HSR achieved a 6.72% and 4.98% improvement over the previous state-of-the-art (Hard-GloVe) on relational bias tasks. By subtracting the predicted gender component, HSR breaks the hidden clusters of biased words that previous methods left untouched.
Semantic Enhancement as Denoising
One of the most striking findings is that debiasing actually improved performance on standard word similarity tasks by nearly 7%, and sentence-level tasks by 14.19%. This suggests that gender bias acts as a form of statistical noise; removing it allows the true semantic relationships between words to emerge more clearly.
Downstream Fairness
On the WinoBias task, HSR reduced the performance gap between pro-stereotypical and anti-stereotypical sentences by 87.26% compared to the runner-up method. This indicates that the model becomes significantly fairer in practical applications like coreference resolution.
Conclusion
HSR represents a shift from geometric to causal debiasing. By treating bias as a confounding variable to be regressed out, the method effectively reduces relational bias while simultaneously improving the overall semantic quality of word embeddings. Future work will focus on extending this framework to contextual embeddings like BERT and GPT and exploring its efficacy against other forms of social bias.
Summary updated