Debiasing Methods in Natural Language Understanding Make Bias More Accessible

Michael Mendelson, Yonatan Belinkov

Abstract: Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debiasing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the discovery of more robust features in the model’s inner representations. We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases from the model’s representations. We experiment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually encoded in its inner representations.

Introduction

Deep learning models are notorious "shortcut learners." In Natural Language Understanding (NLU) tasks, they often latch onto superficial statistical cues rather than actual logic. For example, in Natural Language Inference (NLI), a model might learn that if the premise and hypothesis share many words (lexical overlap), the answer is likely "entailment," regardless of the actual semantic relationship.

Researchers have developed various "debiasing" methods to force models to ignore these shortcuts, typically by training a "bias-only" model and regularizing the main model to deviate from its predictions. While these methods improve performance on "hard" or out-of-distribution (OOD) datasets, the internal state of these models remains under-explored. This paper investigates whether the bias actually vanishes or if the model simply learns a clever workaround.

Probing with Minimum Description Length

The authors propose a probing-based framework to measure how much bias is "extractable" from a model's inner layers. Instead of standard probe accuracy, they utilize Minimum Description Length (MDL). This information-theoretic metric measures how many bits are required to transmit labels given the model's representations. A higher "compression" score ($\mathcal{C}$) indicates that the bias is easier to extract and more prominently encoded.

Probing tasks were designed to detect specific known biases:

  • NegWords: Identifying negation words in the hypothesis (a common cue for "contradiction").
  • Overlap/Subsequence: Detecting if hypothesis words are a subset or a direct subsequence of the premise.

The Robustness Paradox

Counter-intuitively, the study finds that as models become more robust—performing better on OOD challenge sets like HANS—the bias becomes more accessible in their internal representations. This suggests that debiasing does not "clean" the representations. Instead, the model learns to identify biased features more precisely so that the final classification layer can explicitly discount or "subtract" them from the prediction.

Correlation between robustness gains and bias extractability, showing that more robust models often encode bias more clearly.

Key Results

In almost every experiment, debiased models showed higher compression scores for biased features than the "Base" (non-debiased) models. For instance, on the SNLI subsequence bias task, the Base model achieved a compression of 3.2, while debiased versions like DFL jumped significantly higher while simultaneously improving OOD accuracy on HANS.

Performance and compression scores for various debiasing methods. Note that higher HANS- performance (robustness) often correlates with higher compression ($\mathcal{C}$) of the bias.

Key findings include:

  • Strong Correlations: There is a consistent positive Pearson correlation ($\rho$) between a model's OOD improvement and its bias extractability. For lexical overlap on SNLI, this correlation reached 0.752.
  • Fine-tuning Amplifies Bias: Simply fine-tuning a vanilla BERT model on a biased dataset makes the bias more extractable compared to the frozen pretrained version. Debiasing methods further amplify this effect.
  • Linguistic Integrity: Probing for standard linguistic properties (e.g., sentence length) showed that debiased models retain roughly the same amount of general linguistic knowledge as standard ones; the representations are not being degraded, only specialized.

Conclusion

The results indicate that "unbiased predictions imply biased representations." Current debiasing techniques do not make the model "blind" to shortcuts; they make the model an expert at spotting shortcuts so it can avoid them at the decision layer. The authors conclude that current debiasing is a "bad proxy" for learning truly unbiased representations and call for research into methods that encourage the discovery of robust, bias-free feature detectors.

Summary updated

Abstract

Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debiasing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the discovery of more robust features in the model’s inner representations. We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases from the model’s representations. We experiment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually encoded in its inner representations.
BibTeX
@inproceedings{mendelson2021debiasing,
  author = {Michael Mendelson and Yonatan Belinkov},
  title = {{Debiasing Methods in Natural Language Understanding Make Bias More Accessible}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=bivGUFcCsM},
}
Figure