On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations

Roy Schwartz, Gabriel Stanovsky

Deep learning models in Natural Language Processing (NLP) are notorious for exploiting "spurious correlations" or shortcuts. For instance, a model might learn that the presence of the word "not" always implies a "contradiction" label in a logic task, bypassing any actual linguistic understanding of negation. To combat this, researchers typically "balance" datasets by adding counter-examples or using Adversarial Filtering (AF) to remove instances that simple models solve easily.

This position paper argues that such dataset balancing is a losing battle. The authors demonstrate that aggressive filtering eventually removes all learnable signals and that many so-called spurious correlations are actually essential "fallback" meanings that humans use to navigate an underspecified world.

The Futile Arms Race

The current paradigm in NLP benchmark development creates a cycle of diminishing returns. As researchers filter out simple correlations, they build larger models that simply find subtler, more complex shortcuts. This creates an "arms race" where the datasets become increasingly sanitized yet remain unlearnable in a way that reflects true linguistic competence.

The 'arms race' cycle between dataset filtering and model scaling.

This cycle is evident in the evolution of benchmarks like SWAG and HellaSWAG. While baselines on SWAG were initially low (~59%), larger models like BERT quickly reached human-level performance (~86%). This led to the creation of HellaSWAG, which employed more aggressive filtering, only for the cycle to repeat. Filtering does not necessarily improve generalization; it often just makes the shortcuts harder for humans to spot.

Theoretical Limits and Model Overconfidence

The paper provides a theoretical "XOR Argument" to show the limits of filtering. While a dataset can be balanced for single features (unigrams), it remains unbalanced for feature combinations (bigrams). If one were to truly balance for all possible $n$-gram combinations, all statistical signal would eventually be removed, leaving the model with nothing to learn.

Qualitative probing further highlights how models fail when forced to choose. In an experiment with RoBERTa-large fine-tuned on sentiment analysis (SST-2), the model assigned a 99.99% confidence score to a positive sentiment label for the ambiguous sentence: "To my great surprise, the movie turned out different than what I thought."

Current models rely on low-level correlations (like 'great') rather than structural understanding.

Gradient-based interpretations showed the model was almost entirely focused on the words "great" and "surprise," ignoring the structural ambiguity. This "forced choice" inductive bias encourages models to latch onto any available correlation to reduce loss, even when the context is insufficient.

Rethinking "Spurious" Correlations

The authors suggest that some correlations labeled as "spurious" are actually valuable world knowledge. For example, if a model learns that "The President of the U.S." usually refers to the current incumbent, that is a useful fallback for an underspecified question. Drawing on Grice’s Maxims, the paper reminds us that humans only explicitly mention a small fraction of the information needed for understanding; the rest is filled in via common-sense correlations.

Forcing a model to "unlearn" these via balancing can make it less useful in real-world communication. While social-bias correlations (e.g., gender stereotypes) must be mitigated, general world-knowledge fallbacks are a feature of human communication, not a bug.

Proposed Alternatives

Instead of the endless cycle of filtering, the authors propose several shifts in how we train and evaluate models:

  1. Abstention: Developing models that can say "I don't know" or ask for more context when an input is ambiguous rather than forcing a guess.
  2. Richer Contexts: Explicitly training models to handle complex linguistic phenomena like negation, sarcasm, and humor.
  3. Few-Shot Learning: Moving away from massive fine-tuning on biased datasets toward zero- or few-shot setups where models rely more on their pre-trained general knowledge.

Summary updated

Abstract

Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances (Sakaguchi et al., 2020), culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021). In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.
BibTeX
@inproceedings{schwartz2022limitations,
  author = {Roy Schwartz and Gabriel Stanovsky},
  title = {{On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.168},
}
Figure