How far can bias go? - Tracing bias from pretraining data to alignment

Marion Thaler, Abdullatif Köksal, Alina Leidinger, Anna Korhonen, Hinrich SchützearXiv 2024 · DOI · arXiv

  • Tracing bias from pretraining data to alignment

Abstract

Large Language Models (LLMs) frequently exhibit gender biases, yet the lack of transparency in proprietary training sets makes it difficult to pinpoint where these biases originate. This study utilizes the fully open-source OLMo 7B model and its 3-trillion-token Dolma training corpus to trace gender bias from pre-training through supervised fine-tuning (SFT) and instruction alignment. Our results indicate that while alignment can significantly increase the representation of women in model outputs, it acts as a superficial "veneer" that fails to erase the deep-seated stereotypical associations learned during pre-training. The findings suggest that bias mitigation must be addressed at the pre-training stage to be truly effective.

Introduction: The Transparency Gap

The "black box" nature of state-of-the-art models like GPT-4 prevents researchers from determining whether observed biases stem from raw training data, fine-tuning procedures, or specific prompting techniques. By leveraging the OLMo 7B family, this research provides a rare "cradle-to-grave" analysis of bias migration. The study examines how occupational stereotypes in raw web data eventually manifest in the behavior of aligned AI assistants.

Methodology

The researchers employed a multi-stage pipeline to quantify bias:

  1. Dataset Analysis: A co-occurrence analysis of the Dolma corpus across 220 occupations to identify gendered associations.
  2. Model Probing: Evaluation of three model iterations: Base (pre-training), SFT (supervised fine-tuning), and Instruct (human preference alignment).
  3. Robustness Testing: Use of 23 distinct prompt types and various decoding strategies (temperature, top-k/p) to ensure results were not artifacts of specific configurations.
  4. Metrics: The primary metric used was Stereotypical Association (STA), measuring the deviation of model outputs from a neutral distribution.

Results: The Evolution of Representation

The study reveals a dramatic shift in how often women are mentioned as models progress through the alignment pipeline.

Distribution of female representation across the model lifecycle compared to U.S. BLS data and the Dolma training corpus.

As shown in Figure 1, the raw training data (Dolma) and the base model (OLMo 7B) significantly under-represent women compared to U.S. Bureau of Labor Statistics (BLS) ground truth. However, the instruction-tuning phase (OLMo 7B Instruct) causes a massive upward shift in female representation, often exceeding both the training data and real-world demographics. This suggests that alignment successfully addresses the visibility of women but not necessarily the underlying bias.

Key Insights: The Persistence of Stereotypes

Despite the increase in female mentions, the underlying associations remain tethered to the pre-training data.

1. The "Veneer" of Fairness

Instruction tuning acts like a coat of paint. It successfully increases the representation of women, but the associations remain skewed. While the model mentions women more often, it continues to link them to traditionally "female" roles (e.g., nursing, cleaning) while maintaining male associations for STEM and manual labor fields.

2. Correlation and Bias Amplification

Correlation between training data associations and model generations across OLMo 7B (a), SFT (b), and Instruct (c) versions. Source, p. 7

Correlation between training data associations and model generations across OLMo 7B (a), SFT (b), and Instruct (c) versions.

Figure 5 illustrates that the Pearson correlation between biased training data and model output remains moderate to high (0.5 to 0.7) across all stages. Furthermore, base models often act as "bias megaphones," amplifying the under-representation found in the training data rather than merely mirroring it. If women are under-represented in the training data for a specific role, the base model is even less likely to associate women with that role in its output.

3. Robustness of Bias

One of the most significant findings is the "stickiness" of these biases. Changes to hyperparameters like temperature (randomness) or prompt style had no significant effect on the level of gender bias expressed, suggesting that these associations are deeply encoded in the model's weights during the pre-training phase.

Conclusion

The study concludes that attempting to "fix" bias during the alignment phase is insufficient. Alignment provides a superficial correction that masks, rather than removes, the stereotypical associations learned from the "male default" present in massive web-scale datasets. To truly mitigate bias, interventions must occur at the pre-training stage—the source of the "polluted river."

Summary updated

Abstract

As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing on the Dolma dataset and the OLMo model. Using zero-shot prompting and token co-occurrence analyses, we explore how biases in training data influence model outputs. Our findings reveal that biases present in pre-training data are amplified in model outputs. The study also examines the effects of prompt types, hyperparameters, and instruction-tuning on bias expression, finding instruction-tuning partially alleviating representational bias while still maintaining overall stereotypical gender associations, whereas hyperparameters and prompting variation have a lesser effect on bias expression. Our research traces bias throughout the LLM development pipeline and underscores the importance of mitigating bias at the pretraining stage.
BibTeX
@misc{thaler2024how,
  author = {Marion Thaler and Abdullatif Köksal and Alina Leidinger and Anna Korhonen and Hinrich Schütze},
  title = {{How far can bias go? - Tracing bias from pretraining data to alignment}},
  year = {2024},
  eprint = {2411.19240},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19240},
}
Figure