Abstract
Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets. Recent debiasing methods in natural language understanding (NLU) improve performance on such datasets by pressuring models into making unbiased predictions. An underlying assumption behind such methods is that this also leads to the discovery of more robust features in the model's inner representations. We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases from the model's representations. We experiment with several NLU datasets and known biases, and show that, counter-intuitively, the more a language model is pushed towards a debiased regime, the more bias is actually encoded in its inner representations.
Introduction
Deep learning models are notorious "shortcut learners." In Natural Language Inference (NLI) tasks, for example, models often rely on superficial dataset biases—such as assuming that if two sentences share many words, one must entail the other (lexical overlap bias). While researchers have developed various "debiasing" methods to force models to ignore these shortcuts and achieve better out-of-distribution (OOD) generalization, the internal impact of these methods has remained largely unexplored. Does debiasing actually "unlearn" the bias, or does it simply teach the model to suppress that information at the final output stage?
Measuring Bias Extractability
To address this, we propose a general probing-based framework to measure how much bias is encoded in a model's representations. Instead of standard probe accuracy, we utilize an information-theoretic metric: Minimum Description Length (MDL). MDL measures "extractability"—essentially, how many bits of information are needed to transmit the labels given the representations. High compression (low MDL) indicates that the bias is highly accessible and easy for the model to "see" and use.
Experimental Framework
We evaluate three popular debiasing strategies:
- Product of Experts (PoE): Trains a main model to focus on what a separate "bias" model cannot solve.
- Debiased Focal Loss (DFL): Weights the loss function to prioritize "hard" examples that contradict known biases.
- Confidence Regularization (ConfReg): Uses a teacher-student distillation setup to down-weight biased predictions.
These methods were tested across standard NLU benchmarks (SNLI, MultiNLI, and FEVER) and their corresponding challenge sets (HANS and FEVER-Symmetric), which are specifically designed to expose reliance on heuristics like lexical overlap or claim-only biases.
The Robustness-Extractability Paradox
The most surprising finding is a strong positive correlation between a model's robustness (its OOD performance) and how much bias is encoded in its internal layers. To stop using a shortcut in its final predictions, the model seemingly needs to represent that shortcut more clearly.
As shown in Figure 1, there is a clear trend: as models become more robust to the subsequence bias in MNLI, the extractability of that bias from their representations increases ($\rho = 0.626$). This suggests that the model develops a "proxy" for the bias, making it highly accessible so it can actively decide not to rely on it during classification.
The Focusing Effect
This phenomenon is further evidenced by the "focusing effect" in Debiased Focal Loss (DFL). The DFL method uses a focusing parameter ($\gamma$) to control the intensity of the debiasing pressure. As $\gamma$ increases, the model is forced to prioritize examples that are "hard" because they defy the bias.
Figure 2 demonstrates that as the debiasing pressure grows, the extractability of both overlap and subsequence biases rises linearly. This confirms that the more a model is pushed towards a debiased regime, the more concentrated the bias becomes within its internal representations.
Conclusion and Implications
Our results reveal a fundamental distinction between information being present and information being used. Current debiasing methods do not delete "bad" features; instead, they make them highly accessible so the model can actively suppress them.
This has critical implications for social bias. If this trend holds for biases related to gender or race, current debiasing methods might actually be making these social biases more accessible within the model's latent layers, even if the final outputs appear fair. This highlights a need for "representation-robust" methods that don't just fix predictions but result in truly bias-free internal representations.
Summary updated