The Birth of Bias: A case study on the evolution of gender bias in an English language model

Oskar Van Der Wal, Jaap Jumelet, Katrin Schulz, Willem Zuidema2022 · DOI

Abstract

Detecting and mitigating harmful biases in modern language models are widely recognized as crucial, open problems. In this paper, we take a step back and investigate how language models come to be biased in the first place. We use a relatively small language model, using the LSTM architecture trained on an English Wikipedia corpus. With full access to the data and to the model parameters as they change during every step while training, we can map in detail how the representation of gender develops, what patterns in the dataset drive this, and how the model’s internal state relates to the bias in a downstream task (semantic textual similarity). We find that the representation of gender is dynamic and identify different phases during training. Furthermore, we show that gender information is represented increasingly locally in the input embeddings of the model and that, as a consequence, debiasing these can be effective in reducing the downstream bias. Monitoring the training dynamics allows us to detect an asymmetry in how the female and male gender are represented in the input embeddings. This is important, as it may cause naive mitigation strategies to introduce new undesirable biases. We discuss the relevance of the findings for mitigation strategies more generally and the prospects of generalizing our methods to larger language models, the Transformer architecture, other languages and other undesirable biases.

Introduction

Most research into algorithmic bias treats models as "black boxes," focusing on the finished product—measuring how much a pre-trained model like GPT-4 or BERT discriminates. While these evaluations are necessary for safety, they fail to explain the mechanism of bias acquisition. To move beyond "band-aid" solutions, we must understand the "birth" of bias. This paper investigates the training dynamics of an English language model to determine when bias emerges, how it is distributed across the model's architecture, and how these internal states translate to biased performance in downstream tasks.

Methodology: The Glass-Box Approach

To achieve a high-resolution view of bias evolution, we utilized a "glass-box" methodology. We trained a relatively small Long Short-Term Memory (LSTM) model on a standard English Wikipedia corpus. While modern research often focuses on massive Transformers, the smaller scale of the LSTM allowed us to save and analyze the model’s parameters at every single training step.

Our analysis focused on two primary areas:

  1. Internal Mapping: We tracked how gender representations developed within the input embeddings (the model's vocabulary layer) and how internal states shifted over time.
  2. Downstream Evaluation: We measured the impact of these internal changes using Semantic Textual Similarity (STS) benchmarks. This allowed us to quantify how gendered language influenced the model's judgment of whether two sentences shared the same meaning.

Key Findings

1. Bias is Dynamic and Phased

Gender representation does not follow a constant growth curve. Instead, the model undergoes distinct "phases" during training where its understanding of gender shifts significantly. This suggests that the timing of a model's exposure to certain data patterns is as critical as the data itself.

2. The Localization Effect

As training progresses, gender information becomes increasingly "localized" within the input embeddings. This is a vital finding for mitigation: because the bias remains concentrated in the initial embedding layer rather than leaking inextricably into the deeper logic of the model, surgical debiasing of the embedding layer can be highly effective for downstream tasks.

3. The Asymmetry Trap

Perhaps the most significant discovery is that the model does not represent "male" and "female" genders symmetrically. Because the training data (Wikipedia) contains inherent asymmetries in how genders are discussed, the model develops different internal structures for each.

This asymmetry poses a risk for naive mitigation strategies. Many standard debiasing techniques assume gender exists on a simple linear axis; however, applying a symmetrical tool to a lopsided internal structure may inadvertently introduce new, unintended biases.

Discussion and Mitigation

Our results demonstrate that monitoring training dynamics allows for the detection of structural asymmetries that static evaluations miss. By identifying that gender information is represented locally in input embeddings, we show that targeted interventions at the embedding level can successfully reduce bias in tasks like semantic similarity.

However, the presence of asymmetrical encoding serves as a cautionary tale. Mitigation strategies must be informed by the specific way a model has internalized data patterns, rather than assuming a one-size-fits-all mathematical correction.

Conclusion and Future Work

This study provides a high-resolution map of how bias is born and localized during the training of an LSTM. While these findings offer a path toward more effective, surgical debiasing, several questions remain:

  • Scaling: Future research must determine if these "phases of bias" and localization effects hold true for massive Transformer-based models like GPT or Llama.
  • Architectural Differences: The transition from recurrence (LSTM) to attention (Transformers) may alter how bias is distributed.
  • Scope: These tracking methods should be applied to other languages and different forms of bias, such as race, age, or socioeconomic status, to ensure a comprehensive approach to AI fairness.

Summary updated

Abstract

Detecting and mitigating harmful biases in modern language models are widely recognized as crucial, open problems. In this paper, we take a step back and investigate how language models come to be biased in the first place.We use a relatively small language model, using the LSTM architecture trained on an English Wikipedia corpus. With full access to the data and to the model parameters as they change during every step while training, we can map in detail how the representation of gender develops, what patterns in the dataset drive this, and how the model’s internal state relates to the bias in a downstream task (semantic textual similarity).We find that the representation of gender is dynamic and identify different phases during training.Furthermore, we show that gender information is represented increasingly locally in the input embeddings of the model and that, as a consequence, debiasing these can be effective in reducing the downstream bias.Monitoring the training dynamics, allows us to detect an asymmetry in how the female and male gender are represented in the input embeddings. This is important, as it may cause naive mitigation strategies to introduce new undesirable biases.We discuss the relevance of the findings for mitigation strategies more generally and the prospects of generalizing our methods to larger language models, the Transformer architecture, other languages and other undesirable biases.
BibTeX
@misc{wal2022birth,
  author = {Oskar Van Der Wal and Jaap Jumelet and Katrin Schulz and Willem Zuidema},
  title = {{The Birth of Bias: A case study on the evolution of gender bias in an English language model}},
  year = {2022},
  doi = {10.18653/v1/2022.gebnlp-1.8},
}
Figure