On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations

Roy Schwartz, Gabriel Stanovsky

Abstract

Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances, culminating in a recent proposal to eliminate single-word correlations altogether. In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.

The Shortcut Problem in NLP

Deep learning models in Natural Language Processing (NLP) are notorious for being "lazy." Instead of learning the complex nuances of human language, they often rely on spurious correlations: low-level statistical shortcuts between simple features (like a specific word) and a label. For example, a model might learn that the word "not" always implies a "contradiction" label in a logic task, failing miserably when it encounters a sentence where "not" is used differently.

The standard industry response has been dataset balancing: either adding counter-examples or filtering out "easy" instances to force the model to learn "real" features. However, this approach is fundamentally limited and creates a never-ending "arms race" between data curators and increasingly massive models.

The Futility of Balancing: An Endless Arms Race

The paper highlights a recurring cycle in NLP research where new "balanced" datasets are released, only to be "solved" by larger models within a year. This suggests that as we create more aggressive filtering techniques (like AFLite), we simply pave the way for larger models (like GPT-3 or T5) to find even more microscopic correlations that humans cannot easily perceive.

The cycle of dataset filtering and model scaling.

This "arms race" is further illustrated by the XOR analogy: while you can balance a dataset for single features (unigrams), a powerful enough model will simply move up the chain to exploit pairs of features (bigrams). To truly remove all shortcuts, one would eventually have to remove all statistical signals, leaving the model with nothing to learn.

Throwing the Baby Out with the Bathwater

A major risk of perfect balancing is the loss of useful "fallback" options. For humans, if someone asks "Who is the president?", we assume the U.S. president by default unless told otherwise. By "balancing" this out, we prevent models from learning the very world knowledge and common sense we want them to have. Truly balanced datasets might be "clean," but they are also "empty" of the common sense (e.g., "people are happy when they get gifts") necessary for natural communication.

Moving Forward: Alternatives to Balancing

Instead of continuing the lost battle of dataset balancing, the authors propose three primary ways forward:

  1. Richer Contexts: Instead of filtering data, we should enrich it with examples of negation, sarcasm, and metaphors to teach models how context changes meaning.
  2. Abstention and Interaction: Models should be allowed to say "I don't know" or ask for clarification when an input is ambiguous, rather than being forced to make a biased guess based on low-level correlations.
  3. Zero- and Few-Shot Learning: Moving away from massive fine-tuning toward setups where models must rely on their broad pretraining knowledge may naturally reduce their dependency on dataset-specific shortcuts.

To provide a concrete example, the authors tested a RoBERTa-large model on an ambiguous sentiment sentence: "To my great surprise, the movie turned out different than what I thought." Despite the ambiguity, the model predicted "positive" with 99.99% confidence, relying on the shortcut words "great" and "surprise."

Current vs. proposed approaches for handling ambiguous inputs.

By allowing models to interact or abstain, we move away from the current approach of relying on low-level correlations for overconfident predictions on ambiguous text.

Summary updated

Abstract

Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances (Sakaguchi et al., 2020), culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021). In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.
BibTeX
@inproceedings{schwartz2022limitations,
  author = {Roy Schwartz and Gabriel Stanovsky},
  title = {{On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.168},
}
Figure