Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning

Przemyslaw Joniak, Akiko AizawaNAACL 2022

TL;DR

This paper introduces a novel framework using "movement pruning" to pinpoint which parts of transformer-based language models are responsible for gender bias. By selectively disabling parts of the model while fine-tuning it to reduce bias, they can identify "biased" components and even create less biased models with minimal performance loss.

What Problem Does This Paper Address?

Large language models (LLMs) are increasingly used in various applications, but they often inherit and even amplify societal biases present in their training data. Gender bias, specifically, is a well-documented issue where models associate certain professions or traits with particular genders. While many methods exist to debias these models, understanding where this bias is stored within the complex architecture of transformers (like BERT) has been challenging. This lack of granular understanding hinders the development of more efficient and targeted debiasing techniques. The authors aim to fill this gap by developing a method to "prune" or disable specific components of a pre-trained model to see if doing so reduces bias, thereby identifying the biased parts.

Methods

The core of the paper's contribution is a novel framework that combines movement pruning with a debiasing objective.

  • Movement Pruning: Originally used for model compression, movement pruning works by associating a "pruning score" with each weight. During training, these scores are optimized to push weights towards zero. If a weight's score indicates it should be zero, it's effectively pruned.
  • Framework Modification: The authors adapt this by freezing the original model weights and only optimizing the pruning scores. This allows them to identify which weights (and by extension, which model components) can be removed without significantly harming the model's performance on a downstream task, while simultaneously optimizing for a debiasing objective.
  • Targeting Components: They apply this pruning to specific building blocks of transformers, primarily attention heads. They explore two strategies:
  • Square Block Pruning: Pruning square blocks within the matrices of an attention head.
  • Entire Head Pruning: Pruning all matrices associated with an entire attention head.
  • Debiasing Objective: They use the gender bias removal method proposed by Kaneko and Bollegala (2021), which aims to make embeddings of gendered words orthogonal to their stereotypical counterparts.

Datasets

The paper uses existing datasets for evaluation:

  • SEAT (Sentence Encoder Association Test): Measures bias in the embedding space.
  • StereoSet Stereotype Score (SS): Measures bias as perceived by humans, reflecting preferences for stereotypical associations.
  • GLUE (General Language Understanding Evaluation): A benchmark suite of nine tasks used to measure the model's overall performance.

Key Insights and Ideas

  • Bias is Distributed, Not Localized: Contrary to some expectations, pruning entire attention heads didn't always lead to significant bias reduction. This suggests that gender bias might not be confined to a few specific heads but is rather distributed across multiple components of the model.
  • Intermediate Layers Matter Most: The authors found that debiasing intermediate layers of the BERT model was more effective than focusing on the first or last layers. This is visualized in Figure 1, showing density changes across layers.
  • Performance-Bias Trade-off: A clear negative correlation exists: models that perform better on general language understanding tasks (higher GLUE scores) tend to exhibit more gender bias (higher SS scores). This highlights a fundamental challenge in debiasing.
  • Pruning Values Matrix is Key: For attention heads, pruning the "values" matrix (V) is crucial for making the attention output zero, which is essential for effective pruning in this context.

Key Results

  • The proposed framework successfully identifies subsets of the model that contain less bias than the original.
  • They propose new debiasing modes ("intermediate-token" and "intermediate-sentence") that are competitive with existing methods and achieve lower SS scores.
  • The "intermediate-sentence" mode achieved an SS score close to the ideal 50, indicating a significant reduction in gender bias.
  • While performance on most GLUE tasks remained relatively stable or saw minor drops, the model's performance on the COLA (Corpus of Linguistic Acceptability) task suffered significantly when bias was heavily reduced, suggesting a potential trade-off with grammatical reasoning.

Limitations and Future Work

  • Block Size Limitation: Choosing the right square block size for pruning was found to be a limitation, with larger sizes sometimes causing convergence issues.
  • COLA Performance Drop: The significant drop in performance on the COLA task when aggressively debiasing is a notable limitation, suggesting that reducing bias might negatively impact the model's ability to understand complex grammatical structures.
  • Future Work: The authors suggest exploring improvements to the debiasing algorithm, investigating better ways to optimize debiasing and downstream tasks simultaneously, and applying their framework to other types of biases beyond gender. They also hope their framework will find broader applications in analyzing and mitigating biases in language models.

Summary updated

BibTeX
@inproceedings{joniak2022gender,
  author = {Przemyslaw Joniak and Akiko Aizawa},
  title = {{Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure