Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving

Lei Ding, Dengdeng Yu, Jinhan Xie, Wenxing Guo, Shenggang Hu, Meichen Liu, Linglong Kong, Hongsheng Dai, Yanchun Bao, Bei Jiang

Abstract

With widening deployments of natural language processing (NLP) in daily life, inherited social biases from NLP models have become more severe and problematic. Previous studies have shown that word embeddings trained on human-generated corpora have strong gender biases that can produce discriminative results in downstream tasks. Previous debiasing methods focus mainly on modeling bias and only implicitly consider semantic information while completely overlooking the complex underlying causal structure among bias and semantic components. To address these issues, we propose a novel methodology that leverages a causal inference framework to effectively remove gender bias. The proposed method allows us to construct and analyze the complex causal mechanisms facilitating gender information flow while retaining oracle semantic information within word embeddings. Our comprehensive experiments show that the proposed method achieves state-of-the-art results in gender-debiasing tasks. In addition, our methods yield better performance in word similarity evaluation and various extrinsic downstream NLP tasks.

Introduction

Word embeddings like GloVe and Word2Vec serve as the foundational layer for modern Natural Language Processing (NLP). However, these embeddings often inherit toxic social biases present in their training corpora. This leads to problematic associations, such as linking "doctor" predominantly with men and "nurse" with women. While identifying these biases is straightforward, removing them without degrading the model's semantic accuracy remains a significant challenge.

The Challenge of Entangled Bias

Existing debiasing techniques often treat gender and semantic meaning as separate, independent components. In reality, language is deeply entangled. For instance, removing gender information from a word like "bridegroom" might inadvertently strip away its semantic connection to weddings. This research addresses the problem of disentanglement: performing a surgical removal of gender stereotypes while keeping the "oracle" semantic information intact. The authors propose that the underlying causal structure of language must be modeled to achieve this precision.

Methodology: The DeSIP Framework

The researchers introduced DeSIP (Word Embeddings via Causal Inference), a framework that treats word embeddings as a causal system. The method is divided into two primary approaches: P-DeSIP, which utilizes proxy variables, and U-DeSIP, designed for unresolved bias.

P-DeSIP: Handling Proxy Variables

In the P-DeSIP model, the system defines variables for pure gender bias ($D$, represented by difference vectors like $\vec{he} - \vec{she}$), proxy variables ($P$, words with inherent gender like "bride"), and semantic information ($X$). By applying causal inference, the model can mathematically simulate an "intervention" on the proxy variables.

Causal graphs for P-DeSIP showing the original structure and the effect of intervention on proxy variables.

This intervention allows the system to cancel out the influence of bias along specific pathways (e.g., $P \to X \to Y$) while allowing legitimate semantic information to flow. To handle high-dimensional vectors, the authors employ Partial Least Squares (PLS) for parameter estimation.

U-DeSIP: Addressing Unresolved Bias

For cases where bias is not easily captured by explicit proxy variables, the U-DeSIP framework identifies "resolving variables" ($Z$) through heuristic selection based on cosine similarity. This allows for a more flexible debiasing process that can capture subtle, non-explicit gender associations.

Causal graphs for U-DeSIP illustrating the handling of unresolved bias through intervention.

Information Restoration

A critical final step in both methods is the restoration of "residual" information. By calculating the parts of the word vector that are mathematically orthogonal to the gender space, the framework ensures that the non-gendered meaning of the word is preserved and even clarified.

Experimental Evaluation

The researchers evaluated DeSIP using 300-dimensional GloVe embeddings trained on English Wikipedia. Performance was measured across bias benchmarks (WEAT, SemBias), semantic similarity tasks (RG65, SimLex-999), and downstream NLP tasks like Named Entity Recognition (NER).

Key Results

  • State-of-the-Art Bias Reduction: In SemBias tests, P-DeSIP achieved 97.5% accuracy in identifying gender-neutral pairs, significantly outperforming the original GloVe (57.5%) and previous methods like HSR (10%).
  • Enhanced Semantic Integrity: On the RG65 similarity task, P-DeSIP scored 0.7794, surpassing the original GloVe's 0.7540. This suggests that removing the "noise" of social bias actually improves the clarity of word meanings.
  • Downstream Stability: Unlike many debiasing methods that cause performance drops in tasks like NER, DeSIP showed zero degradation and, in some cases, slight improvements in precision.

Conclusion

The DeSIP framework demonstrates that causal inference is a powerful tool for creating fairer AI. By moving beyond simple projections and instead modeling the flow of information, it is possible to surgically remove bias while preserving—and even enhancing—the semantic core of language models. Future work will look toward non-linear causal discovery and applying this framework to other forms of social bias, such as race and class.

Summary updated

Abstract

With widening deployments of natural language processing (NLP) in daily life, inherited social biases from NLP models have become more severe and problematic. Previous studies have shown that word embeddings trained on human-generated corpora have strong gender biases that can produce discriminative results in downstream tasks. Previous debiasing methods focus mainly on modeling bias and only implicitly consider semantic information while completely overlooking the complex underlying causal structure among bias and semantic components. To address these issues, we propose a novel methodology that leverages a causal inference framework to effectively remove gender bias. The proposed method allows us to construct and analyze the complex causal mechanisms facilitating gender information flow while retaining oracle semantic information within word embeddings. Our comprehensive experiments show that the proposed method achieves state-of-the-art results in gender-debiasing tasks. In addition, our methods yield better performance in word similarity evaluation and various extrinsic downstream NLP tasks.
BibTeX
@inproceedings{ding2022word,
  author = {Lei Ding and Dengdeng Yu and Jinhan Xie and Wenxing Guo and Shenggang Hu and Meichen Liu and Linglong Kong and Hongsheng Dai and Yanchun Bao and Bei Jiang},
  title = {{Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21443},
}
Figure