Abstract
Transformer-based models often exhibit indirect bias by relying on context words as proxies for protected attributes, even when explicit identity tokens are removed. This paper proposes a framework for Indirect Bias Discovery (IBD) and Mitigation (IBM). By comparing the attention patterns of a task model against a "helper model" trained to predict protected attributes, the authors identify biased reasoning. A similarity-based regularization term is then introduced to penalize these biased attention patterns during training. Experiments on toxicity and sentiment datasets demonstrate that this approach significantly reduces indirect bias in model explanations with minimal impact on predictive performance.
Introduction
In the field of AI fairness, a distinction is often made between disparate treatment (direct bias) and disparate impact (indirect bias). Direct bias is relatively straightforward to identify and mitigate, such as when a model flags a sentence as hate speech simply because it contains the word "Muslim." This can often be addressed by masking identity tokens.
However, indirect bias is more insidious. Even without explicit identity markers, a model may learn to associate context words like "Quran" or "mosque" with hate speech due to biases in training data. Current fairness metrics frequently focus on the final prediction (outcome) while ignoring the underlying logic (explanation). This paper addresses this gap by discovering and mitigating subtle, indirect associations within model attention weights.
Methodology
The authors propose a two-stage framework to handle indirect bias:
- Indirect Bias Discovery (IBD): A "helper model" ($g$) is trained to predict protected attributes (e.g., gender or race) from text. The attention maps of the main task model ($f$) are then compared to those of the helper model. High similarity suggests the task model is using context words as proxies for protected attributes. This is quantified using the Area Under the Similarity Curve (AUSC), which aggregates Jaccard similarity across various importance thresholds.
- Indirect Bias Mitigation (IBM): To rectify discovered biases, a similarity-based regularization term is added to the loss function. During training, the model is penalized if its attention matrix closely resembles that of the helper model, calculated via squared cosine similarity. This encourages the model to focus on task-relevant ground-truth tokens rather than biased proxies.
Experimental Setup
The framework was evaluated on three major benchmarks:
- Jigsaw Unintended Bias in Toxicity: 2 million public comments evaluated for toxicity versus male/female identities.
- Amazon Books Reviews: A sentiment analysis dataset where gender was inferred from reviewer names.
- Measuring Hate Speech Corpus: Over 50,000 social media comments used to evaluate racial bias.
Key Results and Insights
Prediction vs. Explanation Fairness
A core finding is that traditional methods like Adversarial Debiasing (AD) may achieve demographic parity in predictions but fail to clean up internal attention maps. The model's reasoning remains "tainted" even if the output appears fair.
Quantitative Improvements
On the Jigsaw dataset, the proposed AD+IBM approach reduced the AUSC (indirect bias score) from 0.7067 (Vanilla) to 0.5796. Similar trends were observed across other datasets, where the IBM-enhanced models consistently showed lower similarity to the biased helper model across all token importance levels.
The Utility Trade-off
As with many fairness interventions, there is a slight trade-off between bias mitigation and predictive accuracy. For instance, accuracy on the Jigsaw dataset dropped from approximately 84% to 80%. However, sensitivity analysis allows for finding an optimal regularization parameter ($\lambda$) that balances these competing goals.
Conclusion and Future Work
The IBD/IBM framework provides a dynamic way to handle indirect bias without relying on static keyword lists. While effective, the method's success depends on the quality of the helper model and involves additional computational overhead. Future research will explore extending this approach beyond binary protected attributes and investigating bias in the feed-forward layers of Transformers that follow the self-attention mechanism.
Summary updated