Discovering and Mitigating Indirect Bias in Attention-Based Model Explanations

Farsheed Haque, Depeng Xu, Shuhan YuanNAACL 2024 · DOI

Abstract

Transformer-based models often exhibit indirect bias by relying on context words as proxies for protected attributes, even when explicit identity tokens are removed. This paper proposes a framework for Indirect Bias Discovery (IBD) and Mitigation (IBM). By comparing the attention patterns of a task model against a "helper model" trained to predict protected attributes, the authors identify biased reasoning. A similarity-based regularization term is then introduced to penalize these biased attention patterns during training. Experiments on toxicity and sentiment datasets demonstrate that this approach significantly reduces indirect bias in model explanations with minimal impact on predictive performance.

Introduction

In the field of AI fairness, a distinction is often made between disparate treatment (direct bias) and disparate impact (indirect bias). Direct bias is relatively straightforward to identify and mitigate, such as when a model flags a sentence as hate speech simply because it contains the word "Muslim." This can often be addressed by masking identity tokens.

However, indirect bias is more insidious. Even without explicit identity markers, a model may learn to associate context words like "Quran" or "mosque" with hate speech due to biases in training data. Current fairness metrics frequently focus on the final prediction (outcome) while ignoring the underlying logic (explanation). This paper addresses this gap by discovering and mitigating subtle, indirect associations within model attention weights.

Methodology

The authors propose a two-stage framework to handle indirect bias:

  1. Indirect Bias Discovery (IBD): A "helper model" ($g$) is trained to predict protected attributes (e.g., gender or race) from text. The attention maps of the main task model ($f$) are then compared to those of the helper model. High similarity suggests the task model is using context words as proxies for protected attributes. This is quantified using the Area Under the Similarity Curve (AUSC), which aggregates Jaccard similarity across various importance thresholds.
The IBD framework compares task model f and helper model g to detect biased attention patterns.
  1. Indirect Bias Mitigation (IBM): To rectify discovered biases, a similarity-based regularization term is added to the loss function. During training, the model is penalized if its attention matrix closely resembles that of the helper model, calculated via squared cosine similarity. This encourages the model to focus on task-relevant ground-truth tokens rather than biased proxies.

Experimental Setup

The framework was evaluated on three major benchmarks:

  • Jigsaw Unintended Bias in Toxicity: 2 million public comments evaluated for toxicity versus male/female identities.
  • Amazon Books Reviews: A sentiment analysis dataset where gender was inferred from reviewer names.
  • Measuring Hate Speech Corpus: Over 50,000 social media comments used to evaluate racial bias.

Key Results and Insights

Prediction vs. Explanation Fairness

A core finding is that traditional methods like Adversarial Debiasing (AD) may achieve demographic parity in predictions but fail to clean up internal attention maps. The model's reasoning remains "tainted" even if the output appears fair.

Quantitative Improvements

On the Jigsaw dataset, the proposed AD+IBM approach reduced the AUSC (indirect bias score) from 0.7067 (Vanilla) to 0.5796. Similar trends were observed across other datasets, where the IBM-enhanced models consistently showed lower similarity to the biased helper model across all token importance levels.

Jaccard similarity curves for the Hate Speech dataset show that IBM-based methods significantly lower the area under the curve (AUSC) compared to vanilla models.

The Utility Trade-off

As with many fairness interventions, there is a slight trade-off between bias mitigation and predictive accuracy. For instance, accuracy on the Jigsaw dataset dropped from approximately 84% to 80%. However, sensitivity analysis allows for finding an optimal regularization parameter ($\lambda$) that balances these competing goals.

Conclusion and Future Work

The IBD/IBM framework provides a dynamic way to handle indirect bias without relying on static keyword lists. While effective, the method's success depends on the quality of the helper model and involves additional computational overhead. Future research will explore extending this approach beyond binary protected attributes and investigating bias in the feed-forward layers of Transformers that follow the self-attention mechanism.

Summary updated

BibTeX
@inproceedings{haque2024discovering,
  author = {Farsheed Haque and Depeng Xu and Shuhan Yuan},
  title = {{Discovering and Mitigating Indirect Bias in Attention-Based Model Explanations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.104},
}
Figure