TL;DR
This paper investigates how gender bias is encoded within the internal representations of various pre-trained language models (PLMs). They found that while debiasing techniques often reduce bias in model outputs, they can sometimes fail to address the underlying internal representations, and even inadvertently increase them. This highlights a critical disconnect in current bias mitigation strategies.
What Problem Does This Paper Address?
Pre-trained language models (PLMs) have become foundational to many NLP applications. However, they often inherit and propagate harmful societal biases, particularly gender bias. While much research has focused on identifying and mitigating bias in model outputs, there's a significant gap in understanding how these biases are encoded within the internal representations of PLMs. This is crucial because biases embedded deep within a model's architecture can be difficult to remove and can lead to unfair outcomes, especially with the rise of techniques like retrieval-augmented generation (RAG) that rely heavily on these internal representations. The paper aims to systematically analyze these internal encoding patterns and the effectiveness of various debiasing methods.
Methods
The core methodology employed is Minimum Description Length (MDL) probing. This information-theoretic approach measures how much information about a specific property (in this case, gender) can be compressed from a model's internal representations at different layers.
Here's how it works:
- Probing Datasets: They use datasets where gender information is explicitly available alongside other data (like biographies).
- MDL Calculation: For each layer of a PLM, they calculate the "code length" required to compress the gender information from the layer's representations. A shorter code length means more compressible (and thus, more encoded) gender information.
- Baselines: To distinguish genuine gender encoding from random noise, they compare the compression rates of the actual model against a model initialized with random weights but the same architecture.
- Bias Definition: A model is considered to exhibit gender bias at a layer if the gender information can be compressed significantly more than from a randomly initialized model.
They also analyze the impact of various debiasing techniques (like Counterfactual Data Augmentation, Adapter-Based Debiasing, Dropout, and Orthogonal Projection) and fine-tuning on these compression values.
Datasets
- Bias in Bios dataset: This dataset, containing 396,347 biographies with associated occupations and gender labels, is used for probing. Each entry is a triplet {biography, occupation, gender}.
Key Insights and Ideas
- Consistent Layer-wise Pattern: All tested PLMs exhibit a similar pattern: early layers tend to suppress gender signals, while later layers amplify them, embedding bias deeply into abstract representations.
- Debiasing Techniques' Limited Internal Impact: Surprisingly, many debiasing techniques, while effective at reducing bias in model outputs, show limited success in altering the internal representations. Some even inadvertently increase the encoded bias in these internal layers.
- Pretraining vs. Post-hoc Debiasing: Pretrained debiasing objectives (like training-time CDA) are generally more effective than post-hoc methods in reducing encoded bias.
- Fine-tuning as Implicit Debiasing: Task-specific fine-tuning (e.g., for occupation prediction) consistently reduces encoded gender information, even falling below random baselines. This suggests fine-tuning might actively suppress gender information in favor of task-relevant features.
- Bias Concentration in Classification Heads: The reduction in internal representation bias after fine-tuning suggests that residual bias might be concentrated in the classification layers added during fine-tuning, rather than the encoder itself.
Key Results
- Layer-wise Compression: Larger models (BERT-large) show higher compression from the first layer compared to smaller models (BERT-base). All models show a peak in gender information compression in their final layers.
- Debiasing Effectiveness: Table 1 shows that while debiased models generally perform better on intrinsic bias benchmarks (CrowS-Pairs, StereoSet, DiFair), the paper's internal analysis (Figures 1-3) reveals that many methods don't significantly reduce internal gender encoding. Training-time CDA on BERT-large showed the most substantial reduction in final-layer compression (from 23.08 to 11.98).
- Fine-tuning Impact: Figure 4 shows that fine-tuning consistently reduces gender information compression across layers, often falling below random baselines. For example, fine-tuned BERT-base had significantly lower compression than its pretrained version.
Limitations and Future Work
- Scope of Bias: The study focuses solely on gender bias in English biographies, leaving broader sociocultural and intersectional biases unexplored.
- Model Architectures: The analysis is limited to encoder-based models; future work should extend to decoder-based and multimodal systems.
- Interplay of Fine-tuning and Bias: The complex interplay between task-specific fine-tuning and bias propagation requires deeper exploration across diverse applications.
- Holistic Mitigation: The findings advocate for a paradigm shift towards inherently fair model architectures and training frameworks, integrating fairness objectives during pretraining and targeting bias propagation pathways, rather than relying solely on post-hoc adjustments. Future work could explore hybrid strategies combining training-time and post-hoc techniques.
Summary updated