A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings

Chris Sweeney, Maryam NajafianACL 2019 · DOI

Abstract

This paper introduces Relative Negative Sentiment Bias (RNSB), a new framework and metric designed to measure demographic bias in word embeddings. Unlike traditional geometric metrics that are difficult to interpret, RNSB quantifies bias by measuring the probability that a sentiment classifier assigns negative sentiment to specific identity terms. This approach provides a direct link between embedding-level bias and unfairness in downstream Natural Language Processing (NLP) applications.

Introduction

Word embeddings are mathematical representations of words used extensively in modern NLP. However, these embeddings often "bake in" societal prejudices found in their training data—for instance, associating "doctor" more closely with "man" than "woman."

Existing tools like the Word Embedding Association Test (WEAT) rely on complex vector-space geometry. While effective at proving bias exists, they fail to demonstrate how that bias manifests in real-world applications, such as sentiment analysis. Additionally, many tests are limited to binary comparisons (e.g., "man vs. woman"), making it difficult to evaluate broad categories like national origin or religion across multiple identities simultaneously.

Methodology: The RNSB Framework

The authors propose a transparent framework that moves from pure geometry toward task-based evaluation. The process consists of three primary steps:

  1. Training a Probe: A pre-trained word embedding (e.g., GloVe) is used to train a simple logistic regression classifier on a standard sentiment dataset (e.g., the Hu and Liu lexicon).
  2. Inference on Identity Terms: Neutral identity terms (e.g., "Mexican," "Irish," "Christian") are fed into the classifier. In a fair model, these terms should receive an equal, low probability of being classified as "negative."
  3. Quantifying Bias: The framework uses Kullback–Leibler (KL) divergence to compare the actual distribution of negative sentiment probabilities against a perfectly fair uniform distribution.
The RNSB evaluation framework pipeline, illustrating the transition from biased embeddings to negative sentiment analysis.

Experimental Results

The study evaluated three major embedding models—GloVe, Word2vec, and ConceptNet—across case studies involving National Origin and Religion.

Case Study GloVe (Bias) Word2Vec (Bias) ConceptNet (Bias)
National Origin 0.6225 0.1945 0.0102
Religion 0.3692 0.1026 0.0291

As shown in the table and visualizations below, GloVe was consistently the most biased, while ConceptNet achieved the lowest bias scores. This suggests that incorporating curated knowledge, rather than relying solely on raw web crawls, significantly impacts model fairness.

Negative sentiment distributions for GloVe and ConceptNet compared to a fair uniform distribution.

Discussion and Insights

The RNSB metric reveals a "clumping" effect where biased embeddings unfairly entangle certain groups with negative sentiment. For example, in GloVe, certain nationalities received nearly double the negative sentiment probability compared to the fair baseline. Because RNSB can evaluate dozens of identities simultaneously, it provides a more holistic view of a model's prejudices than previous methods.

Conclusion and Future Work

The RNSB framework provides a scalable "thermometer" for measuring bias in NLP systems. While this study focused on negative sentiment and binary classification, future work could expand the framework to positive sentiment and more complex multi-class tasks. By providing a transparent metric, the authors enable practitioners to better evaluate and choose base models to minimize downstream discrimination.

Summary updated

BibTeX
@inproceedings{sweeney2019transparent,
  author = {Chris Sweeney and Maryam Najafian},
  title = {{A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1162},
}
Figure