Stubborn Lexical Bias in Data and Models

Sofia Serrano, Jesse Dodge, Noah A. Smith

Abstract: In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior. However, the presence and effect of such correlations are typically examined feature by feature. We investigate the cumulative impact on a model of many such intersecting features. Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data. We select two tasks— natural language inference and duplicate-question detection— for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment. The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels. We then apply an optimization approach to reweight the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams). We close with discussion about the implications of our results on what it means to “debias” training data, and how issues of data quality can affect model bias.

The Challenge of Spurious Correlations

Large-scale NLP datasets are often uncurated, leading to unintended associations between specific words (lexical features) and labels. For instance, in the SNLI dataset, the word "nobody" is a near-perfect predictor of the "contradiction" label, regardless of the actual logic of the sentences. While researchers have long identified these "shortcuts," they are typically studied in isolation. This research addresses the cumulative impact of thousands of intersecting biases and tests the common assumption that balancing these associations in the data will eliminate model bias.

Measuring and Mitigating Bias

To rigorously evaluate model bias, the authors introduce a robust framework based on a permutation test. This method splits test instances into "usual-label" sets (where word-label pairings match training biases) and "unusual-label" sets. A model is considered biased if it demonstrates significantly higher accuracy on the "usual" set compared to the "unusual" one.

The permutation test methodology used to identify if models rely on spurious lexical correlations.

To mitigate these biases, the study employs an optimization-based reweighting strategy. Rather than deleting examples, they assign a weight ($q_i$) to every training instance to make the label distribution for thousands of unigrams as uniform as possible. This aggressive data balancing was tested on high-capacity models like RoBERTa-large and simpler LSTMs trained from scratch to isolate the influence of pretraining.

The Persistence of Bias

Despite comprehensive reweighting, the results reveal that lexical bias is remarkably stubborn. For models trained on standard data, the permutation test yielded $p$-values as low as $10^{-35}$ for SNLI, indicating overwhelming evidence of bias. Surprisingly, after reweighting the data to be mathematically "fairer," the $p$-values remained vanishingly small (e.g., $10^{-14}$ for SNLI), suggesting the model still found ways to exploit the underlying data patterns.

Statistical significance (p-values) of bias across different models and datasets, showing that bias persists even after data reweighting.

The "Whack-a-Mole" Effect

A striking finding is that fixing unigram biases (single words) often worsened the bias for bigrams (two-word sequences). In SNLI, the average gap between the label distribution and a uniform distribution for bigrams increased from 0.059 to 0.122 after unigram reweighting. By forcing the model to ignore single-word shortcuts, it simply looked "deeper" into the data to find more complex versions of the same spurious correlations.

Discussion and Future Directions

The persistence of bias in LSTMs trained from scratch suggests that the tendency to take shortcuts is inherent to how these models learn, not just a side effect of pretraining. Furthermore, "unusual" examples—those that defy the bias—might be of lower quality or harder for models to generalize from, meaning simple reweighting is insufficient to force logical reasoning.

The core conclusion is that bias reduction cannot be treated purely as a "data problem." Future work must focus on model architectures and training objectives that explicitly discourage shortcut learning, as it is mathematically impossible to balance all possible feature combinations simultaneously.

Summary updated

Abstract

In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior. However, the presence and effect of such correlations are typically examined feature by feature. We investigate the cumulative impact on a model of many such intersecting features. Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data. We select two tasks— natural language inference and duplicate-question detection— for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment. The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels. We then apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams). We close with discussion about the implications of our results on what it means to “debias” training data, and how issues of data quality can affect model bias.
BibTeX
@inproceedings{serrano2023stubborn,
  author = {Sofia Serrano and Jesse Dodge and Noah A. Smith},
  title = {{Stubborn Lexical Bias in Data and Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=dRl7UqWrBT},
}
Figure