Deep learning models are notoriously opportunistic. If a dataset contains a superficial statistical pattern—a "shortcut"—that correlates with the target label, a neural network will almost always exploit it instead of learning the deeper, more complex underlying logic.
For example, as noted by Jie Ma et al. in Robust Visual Question Answering: Datasets, Methods, and Future Challenges, a Visual Question Answering (VQA) model might learn to answer "yellow" whenever it reads the phrase "What color is the...", completely ignoring the accompanying image. Similarly, Sara Rajaee demonstrates in Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference that Natural Language Inference (NLI) models frequently rely on simple word-overlap heuristics between sentences to predict entailment.
When these models are evaluated on out-of-distribution (OOD) datasets where the shortcuts no longer align with the true labels, their performance collapses. To build truly robust models, we need a way to prevent them from taking the easy way out.
The Core Intuition: Outsourcing the Bias
The core intuition behind residual-based debiasing is elegant: if you want a primary model to ignore easy shortcuts, you should build a secondary model whose only job is to learn them.
Instead of trying to manually clean a dataset or engineer complex features, this approach introduces an intentionally limited "bias-only" or "weak" model alongside the primary model during training. For instance, in a VQA task, the bias-only model might be given access only to the text of the question, forcing it to rely entirely on language priors.
By training the primary model in an ensemble with this biased proxy, we can mathematically "factor out" the shortcuts, forcing the primary model to focus exclusively on the remaining, more complex features (such as actual visual grounding).
The Gradient Mechanics: Product of Experts vs. Residual Fitting
To understand why this works, we must look at how gradients flow during training. There are two primary formulations for combining the main model and the biased model: Product of Experts (PoE) and Residual Fitting.
Product of Experts
In a PoE framework, popularized in this domain by Christopher Clark et al. in Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases, the final ensemble probability is the product of the probabilities from both models (or the sum of their unnormalized log-probabilities):
$$\log P_{\text{ensemble}} = \log P_{\text{bias}} + \log P_{\text{main}}$$
When an example can be easily solved using a dataset shortcut, the bias-only model outputs a high-confidence, correct prediction ($P_{\text{bias}} \approx 1$). Because the ensemble's overall loss on this example is already minimized by the biased expert, the gradient signal sent to the main model drops to near zero. The main model is effectively told: "The shortcut model has this covered; do not waste your capacity learning this pattern." The main model is only forced to update its weights when the shortcut fails, shifting its learning focus toward robust, generalizable features.
Residual Fitting
Alternatively, the problem can be framed as training the main model to fit the residual errors of the biased model. As implemented by He He et al. in Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual, the final prediction is treated as an additive combination of a fixed biased model and a trainable main model.
If the biased model correctly predicts the label of an instance, the main model receives no gradient update. This strategy has proven highly effective across diverse domains; for example, Kijong Han et al. leverage it in An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model to prevent dialogue systems from over-relying on keyword-overlap shortcuts.
Discovering Biases Automatically
Early iterations of these methods required researchers to know and explicitly model the dataset biases beforehand (e.g., designing a question-only model for VQA). However, modern frameworks have generalized this mechanism to handle unknown biases.
The breakthrough realization, articulated by Victor Sanh et al. in Learning from others' mistakes: Avoiding dataset biases without modeling them, is that shortcuts are statistically simple by definition. Therefore, a generic, capacity-limited "weak learner" (such as a shallow neural network) will naturally latch onto shortcuts before it can learn complex logic. By training a main model in a PoE ensemble with this automatically trained weak learner, the main model is shielded from whatever simple shortcuts exist in the data, without requiring any manual bias definition.
Taking this a step further, Prasetya Ajie Utama et al. introduce a self-debiasing framework in Towards Debiasing NLU Models from Unknown Biases. Their approach dynamically identifies examples where a bias-prone auxiliary model exhibits high confidence, and subsequently down-weights those examples in the primary model's loss function, successfully mitigating unknown superficial correlations.
Where Does the Bias Go?
An intuitive assumption might be that residual-based training completely purges the main model's internal representations of any biased features. However, the reality is far more interesting.
In Debiasing Methods in Natural Language Understanding Make Bias More Accessible, Michael Mendelson et al. uncover a striking paradox: models trained via ensemble-based debiasing actually encode dataset biases more clearly within their hidden layers than standard models do.
The ensemble mechanism does not force the primary model to become blind to the shortcut. Instead, it forces the model to learn a highly distinct representation of the shortcut so that it can actively suppress it at the final decision layer. The model doesn't forget the bias; it learns exactly what the bias looks like so it can choose to ignore it at the finish line.
Key Foundational Work
Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019) — contributed 4 concept elements to this direction.
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (2019) — contributed 3 concept elements to this direction.
Look at the First Sentence: Position Bias in Question Answering (2020) — contributed 2 concept elements to this direction.