Abstract: Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debiasing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the discovery of more robust features in the model’s inner representations. We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases from the model’s representations. We experiment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually encoded in its inner representations.
Introduction
Deep learning models are notorious "shortcut learners." In Natural Language Understanding (NLU) tasks, they often latch onto superficial statistical cues rather than actual logic. For example, in Natural Language Inference (NLI), a model might learn that if the premise and hypothesis share many words (lexical overlap), the answer is likely "entailment," regardless of the actual semantic relationship.
Researchers have developed various "debiasing" methods to force models to ignore these shortcuts, typically by training a "bias-only" model and regularizing the main model to deviate from its predictions. While these methods improve performance on "hard" or out-of-distribution (OOD) datasets, the internal state of these models remains under-explored. This paper investigates whether the bias actually vanishes or if the model simply learns a clever workaround.
Probing with Minimum Description Length
The authors propose a probing-based framework to measure how much bias is "extractable" from a model's inner layers. Instead of standard probe accuracy, they utilize Minimum Description Length (MDL). This information-theoretic metric measures how many bits are required to transmit labels given the model's representations. A higher "compression" score ($\mathcal{C}$) indicates that the bias is easier to extract and more prominently encoded.
Probing tasks were designed to detect specific known biases:
- NegWords: Identifying negation words in the hypothesis (a common cue for "contradiction").
- Overlap/Subsequence: Detecting if hypothesis words are a subset or a direct subsequence of the premise.
The Robustness Paradox
Counter-intuitively, the study finds that as models become more robust—performing better on OOD challenge sets like HANS—the bias becomes more accessible in their internal representations. This suggests that debiasing does not "clean" the representations. Instead, the model learns to identify biased features more precisely so that the final classification layer can explicitly discount or "subtract" them from the prediction.
Key Results
In almost every experiment, debiased models showed higher compression scores for biased features than the "Base" (non-debiased) models. For instance, on the SNLI subsequence bias task, the Base model achieved a compression of 3.2, while debiased versions like DFL jumped significantly higher while simultaneously improving OOD accuracy on HANS.
Key findings include:
- Strong Correlations: There is a consistent positive Pearson correlation ($\rho$) between a model's OOD improvement and its bias extractability. For lexical overlap on SNLI, this correlation reached 0.752.
- Fine-tuning Amplifies Bias: Simply fine-tuning a vanilla BERT model on a biased dataset makes the bias more extractable compared to the frozen pretrained version. Debiasing methods further amplify this effect.
- Linguistic Integrity: Probing for standard linguistic properties (e.g., sentence length) showed that debiased models retain roughly the same amount of general linguistic knowledge as standard ones; the representations are not being degraded, only specialized.
Conclusion
The results indicate that "unbiased predictions imply biased representations." Current debiasing techniques do not make the model "blind" to shortcuts; they make the model an expert at spotting shortcuts so it can avoid them at the decision layer. The authors conclude that current debiasing is a "bad proxy" for learning truly unbiased representations and call for research into methods that encourage the discovery of robust, bias-free feature detectors.
Summary updated