Gender Bias in BERT - Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task

Sophie Jentzsch, Cigdem TuranNAACL 2022

TL;DR

This paper introduces a novel method to measure gender bias in BERT models by analyzing sentiment classification. They found that most BERT models exhibit significant gender biases, and these biases primarily originate from the pre-trained models themselves rather than the downstream task data.

What Problem Does This Paper Address?

As powerful language models like BERT become integral to many applications, understanding and mitigating their inherent biases is crucial. Previous methods for bias detection, often developed for static word embeddings, are not easily transferable to contextualized models like BERT. This paper addresses the challenge of quantifying gender bias in BERT within a realistic downstream task (sentiment analysis of movie reviews) and investigates the sources of this bias. The authors highlight that biases in these models can lead to real-world harm, such as unfair recommendations or reinforcing societal stereotypes.

Methods

The authors propose a novel bias measurement technique that leverages the sentiment classification capability of BERT models. Instead of relying on external association tests, they create male and female versions of test samples by masking gendered terms. The sentiment classifier then rates these versions, and the difference in ratings quantifies the bias.

The experimental pipeline involved:

  1. Data Preparation: Manipulating training data by either removing gender terms or creating balanced datasets with both male and female versions of samples.
  2. Classifier Training: Fine-tuning seven different pre-trained BERT models (DistilBERT, BERT base/large, RoBERTa base/large, ALBERT base/large) on the prepared IMDB movie review dataset.
  3. Data Rating: Using the fine-tuned classifiers to rate the masked male and female versions of test samples.
  4. Analysis: Calculating the bias based on the sentiment rating differences.

They compared 63 different classifier configurations resulting from variations in the pre-trained model, training data manipulation, and the set of gender terms used for masking.

Datasets

The primary dataset used was the IMDB Large Movie Review Dataset, consisting of 50,000 movie reviews. This dataset was pre-processed by removing punctuation and converting text to lowercase. The authors also mention using the BookCorpus and English Wikipedia as the original training data for the pre-trained BERT models.

Key Insights and Ideas

  • Sentiment Classifiers as Bias Detectors: The core innovation is using a sentiment classifier's inherent valuation ability to directly measure bias, avoiding the need for separate association tests.
  • Pre-trained Models are the Primary Source of Bias: The study strongly suggests that the biases observed in the downstream sentiment classification task originate from the underlying pre-trained BERT models, not from the task-specific IMDB data.
  • Model Architecture and Size Matter: Larger models and certain architectures (e.g., RoBERTa, ALBERT) tend to exhibit greater gender biases than smaller or different architectures (e.g., DistilBERT).
  • Term Set Size Influences Bias Magnitude: Using larger sets of gendered terms for masking generally leads to larger measured biases, indicating that biases are encoded across a wide range of vocabulary.
  • Task-Specific Data Manipulation Has Limited Impact: Removing or balancing gender terms in the training data had a surprisingly small effect on reducing the final classifier bias, reinforcing the idea that the bias is deeply embedded in the pre-trained models.

Key Results

  • Ubiquitous Bias: Out of 63 experimental models, 57 showed highly significant gender biases.
  • Direction of Bias: While 41 classifiers preferred male terms over female terms, 16 preferred female terms over male terms, indicating that biases are directed differently across models and conditions.
  • Model Size Correlation: Larger models within the same architecture consistently showed higher biases (e.g., BERT-large vs. BERT-base).
  • Architecture Impact: DistilBERT showed about half the bias of BERT and RoBERTa, while ALBERT showed about twice the bias of BERT and RoBERTa.
  • Term Set Impact: Biases were smallest for the pronoun-only set and largest for the comprehensive "all" term set.

Limitations and Future Work

  • Binary Gender Assumption: The study acknowledges that its approach assumes a binary gender classification and does not capture the full spectrum of gender diversity.
  • Term Mapping Ambiguity: Constructing the gendered term lists involved subjective decisions and potential ambiguities in mapping terms.
  • Limited Scope: The investigation focused on gender bias in sentiment classification. Future work could explore other bias dimensions and downstream tasks.
  • Debiased Models: The authors suggest conducting similar experiments on debiased BERT models to test the effectiveness of debiasing techniques.
  • Non-Binary Classification: Adapting the bias measurement approach to non-binary classification tasks is proposed as an exciting future direction.
  • Contextual Nuances: Further investigation into the specific contexts and types of individual samples that lead to contrary bias directions is recommended.

Summary updated

BibTeX
@inproceedings{jentzsch2022gender,
  author = {Sophie Jentzsch and Cigdem Turan},
  title = {{Gender Bias in BERT - Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure