Abstract
In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior. However, the presence and effect of such correlations are typically examined feature by feature. We investigate the cumulative impact on a model of many such intersecting features. Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data. We select two tasks — natural language inference and duplicate-question detection — for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment. The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels. We then apply an optimization approach to reweight the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams). We close with discussion about the implications of our results on what it means to "debias" training data, and how issues of data quality can affect model bias.
Introduction
Natural Language Processing (NLP) models are notorious for taking "shortcuts." Instead of learning the underlying logic of a task, they often latch onto spurious correlations—lexical biases—present in the training data. A classic example is a model associating the word "nobody" with the label "contradiction" in Natural Language Inference (NLI) tasks, regardless of the actual context.
While previous research has identified these biases on a case-by-case basis, this study investigates the cumulative impact of thousands of intersecting features. The central question is whether we can "debias" training data in bulk through mathematical reweighting, and if doing so forces high-capacity models to learn the intended task logic rather than simply finding new shortcuts.
Methodology
Measuring Model Bias: The Permutation Test
To rigorously detect bias, the authors developed a statistical framework using a permutation test. They compared model accuracy on "usual-label" instances (where a word and label align with known training biases) against "unusual-label" instances. This provides a robust metric to determine if a performance gap is statistically significant, even when working with noisy, held-out data.
Data Intervention: Constrained Optimization
Rather than manually filtering or augmenting data, the researchers treated debiasing as a constrained optimization problem. They assigned a weight ($q$) to every training instance, aiming to find a distribution where the label distribution for thousands of different unigrams became as uniform as possible. This effectively "neutralizes" the statistical signal of these words at the data level.
Model Probing
The study evaluated both RoBERTa-large (a high-capacity pretrained model) and LSTMs trained from scratch. By comparing these, the researchers could determine if observed biases were a result of pretraining or inherent to the data distribution itself.
Experimental Setup
The study focused on tasks where individual words (unigrams) should ideally be uninformative:
- Natural Language Inference (SNLI, MNLI, QNLI): Determining if a premise entails or contradicts a hypothesis.
- Quora Question Pairs (QQP): Detecting if two questions are duplicates.
Key Findings
1. The "Whack-a-Mole" Effect
The most significant discovery was that reweighting data to eliminate unigram bias caused the models to migrate their reliance to more complex features. Specifically, bias in bigrams (two-word sequences) actually increased. For example, in the SNLI dataset, the bigram bias rose from 0.059 to 0.122 after unigram reweighting. The model essentially traded one shortcut for another.
2. Stubborn Model Bias
Despite the optimization successfully balancing the training data (e.g., reducing QQP error from 0.154 to 0.047), the trained models remained heavily biased. Permutation tests yielded $p$-values ranging from $10^{-14}$ to $10^{-762}$, providing overwhelming evidence that the models were still prioritizing the original "usual" labels despite the reweighted distribution.
3. Pretraining is Not the Sole Culprit
While one might blame the massive pretraining of RoBERTa for these biases, LSTMs trained from scratch on the reweighted data showed even stronger evidence of bias. This suggests that the signal for these shortcuts remains deeply embedded in the data distribution, and high-capacity models are exceptionally efficient at extracting it.
Discussion and Conclusion
The results suggest that lexical bias in NLP cannot be treated purely as a "data problem" solvable by simple reweighting or filtering. High-capacity models are so adept at finding patterns that they will bypass unigram constraints to find shortcuts in higher-order features like bigrams.
This "stubbornness" implies that future debiasing efforts must move beyond data curation and focus on model-side interventions. Changing how models represent information or how they are incentivized to learn may be the only way to truly break their reliance on spurious correlations. As long as the underlying data distribution contains these signals, models will likely find a way to exploit them.
Summary updated