- Tracing bias from pretraining data to alignment
Abstract
Large Language Models (LLMs) frequently exhibit gender biases, yet the lack of transparency in proprietary training sets makes it difficult to pinpoint where these biases originate. This study utilizes the fully open-source OLMo 7B model and its 3-trillion-token Dolma training corpus to trace gender bias from pre-training through supervised fine-tuning (SFT) and instruction alignment. Our results indicate that while alignment can significantly increase the representation of women in model outputs, it acts as a superficial "veneer" that fails to erase the deep-seated stereotypical associations learned during pre-training. The findings suggest that bias mitigation must be addressed at the pre-training stage to be truly effective.
Introduction: The Transparency Gap
The "black box" nature of state-of-the-art models like GPT-4 prevents researchers from determining whether observed biases stem from raw training data, fine-tuning procedures, or specific prompting techniques. By leveraging the OLMo 7B family, this research provides a rare "cradle-to-grave" analysis of bias migration. The study examines how occupational stereotypes in raw web data eventually manifest in the behavior of aligned AI assistants.
Methodology
The researchers employed a multi-stage pipeline to quantify bias:
- Dataset Analysis: A co-occurrence analysis of the Dolma corpus across 220 occupations to identify gendered associations.
- Model Probing: Evaluation of three model iterations: Base (pre-training), SFT (supervised fine-tuning), and Instruct (human preference alignment).
- Robustness Testing: Use of 23 distinct prompt types and various decoding strategies (temperature, top-k/p) to ensure results were not artifacts of specific configurations.
- Metrics: The primary metric used was Stereotypical Association (STA), measuring the deviation of model outputs from a neutral distribution.
Results: The Evolution of Representation
The study reveals a dramatic shift in how often women are mentioned as models progress through the alignment pipeline.
As shown in Figure 1, the raw training data (Dolma) and the base model (OLMo 7B) significantly under-represent women compared to U.S. Bureau of Labor Statistics (BLS) ground truth. However, the instruction-tuning phase (OLMo 7B Instruct) causes a massive upward shift in female representation, often exceeding both the training data and real-world demographics. This suggests that alignment successfully addresses the visibility of women but not necessarily the underlying bias.
Key Insights: The Persistence of Stereotypes
Despite the increase in female mentions, the underlying associations remain tethered to the pre-training data.
1. The "Veneer" of Fairness
Instruction tuning acts like a coat of paint. It successfully increases the representation of women, but the associations remain skewed. While the model mentions women more often, it continues to link them to traditionally "female" roles (e.g., nursing, cleaning) while maintaining male associations for STEM and manual labor fields.
2. Correlation and Bias Amplification
Correlation between training data associations and model generations across OLMo 7B (a), SFT (b), and Instruct (c) versions. Source, p. 7
Figure 5 illustrates that the Pearson correlation between biased training data and model output remains moderate to high (0.5 to 0.7) across all stages. Furthermore, base models often act as "bias megaphones," amplifying the under-representation found in the training data rather than merely mirroring it. If women are under-represented in the training data for a specific role, the base model is even less likely to associate women with that role in its output.
3. Robustness of Bias
One of the most significant findings is the "stickiness" of these biases. Changes to hyperparameters like temperature (randomness) or prompt style had no significant effect on the level of gender bias expressed, suggesting that these associations are deeply encoded in the model's weights during the pre-training phase.
Conclusion
The study concludes that attempting to "fix" bias during the alignment phase is insufficient. Alignment provides a superficial correction that masks, rather than removes, the stereotypical associations learned from the "male default" present in massive web-scale datasets. To truly mitigate bias, interventions must occur at the pre-training stage—the source of the "polluted river."
Summary updated