The Effect of Natural Distribution Shift on Question Answering Models

John Miller, Karl Krauth, Benjamin Recht, Ludwig Schmidt

TL;DR

Researchers replicated the SQuAD dataset creation process to build four new test sets, finding that while models haven't "overfit" to the original leaderboard, they are surprisingly fragile. While performance holds up on new Wikipedia articles, it plummets by up to 17 F1 points when moving to natural domains like Reddit or Amazon reviews—shifts that humans handle with ease.

What Problem Does This Paper Address?

In the world of Natural Language Processing (NLP), the Stanford Question Answering Dataset (SQuAD) is a gold standard. Since 2016, model performance has soared, with top models now outperforming humans on the official leaderboard. However, two nagging questions remain:

  1. Adaptive Overfitting: Because the SQuAD test set is used repeatedly to rank models, are researchers inadvertently "tuning" their models to the specific quirks of that one test set?
  2. Robustness to Distribution Shift: SQuAD is based entirely on Wikipedia. If we take a model trained on Wikipedia and ask it questions about a New York Times article or a Reddit thread, does it still work?

This paper seeks to determine if the "progress" we see on leaderboards represents a genuine leap in machine understanding or if we are simply building highly specialized engines that stall the moment they leave the "Wikipedia track."

Methods

The authors didn't build a new model; instead, they built a more rigorous "stress test." They meticulously replicated the original SQuAD data collection pipeline:

  • Passage Curation: They selected new text from four sources.
  • Crowdsourcing: They used Amazon Mechanical Turk to have humans write questions and provide answers, following the exact protocols used by the original SQuAD creators.
  • Evaluation: They evaluated over 100 existing models from the SQuAD leaderboard on these new sets to see how their performance translated.

By keeping the process the same and only changing the source text, they created a controlled experiment to measure the impact of "natural distribution shifts."

Datasets

The study utilizes the original SQuAD v1.1 as a baseline and introduces four new test sets:

  • New Wikipedia: Articles from the same timeframe as the original SQuAD but not included in the original test set.
  • New York Times: Professional journalism, representing a slight shift in style.
  • Reddit: Highly informal, conversational text with varying grammar.
  • Amazon Product Reviews: Subjective, opinionated text focused on consumer goods.

Key Insights and Ideas

The results reveal a stark contrast between in-domain generalization and out-of-domain robustness.

Model performance across new domains compared to original SQuAD
  • The Holdout Method is Resilient: One of the most surprising findings is the absence of adaptive overfitting. Models performed almost as well on "New Wikipedia" as they did on the original test set. This suggests that the standard practice of using a fixed holdout set for leaderboards is more robust than many theorists feared.
  • The "Robustness Gap": While models are great at new Wikipedia articles, they struggle with "natural" shifts. The drop in performance is nearly linear: as the text becomes less like Wikipedia (e.g., Reddit), the models fail more frequently.
  • Humans are the Gold Standard for Robustness: The researchers established a human baseline. Unlike the models, humans showed almost no performance drop when moving from Wikipedia to Reddit or Amazon. This highlights a fundamental difference between how humans "understand" text and how current QA models "process" it.
  • Leaderboard Rank is (Mostly) Stable: Interestingly, the best model on SQuAD is usually still the best model on Reddit, but its absolute performance is much lower. There is a "performance tax" for changing domains that hits all models almost equally.

Key Results

The researchers measured performance using the F1 score (a measure of overlap between the predicted and ground-truth answers):

  • New Wikipedia: Models saw a negligible average drop of 1.5 F1 points.
  • New York Times: A modest drop of 3.8 F1 points.
  • Reddit: A significant drop of 14.0 F1 points.
  • Amazon Reviews: A massive drop of 17.4 F1 points.

In contrast, the human baseline only dropped by about 3.0 F1 points even on the most difficult Amazon dataset, maintaining an F1 score of 92.1 compared to the top model's (XLNet) 81.7.

Limitations and Future Work

The authors acknowledge that they still don't fully understand why these shifts are so hard for models. They tested common theories—such as answer diversity or syntactic complexity—but found that these factors only explained a tiny fraction of the performance drop.

Future work is needed to:

  1. Develop better metrics that explicitly incorporate robustness into evaluation.
  2. Investigate training techniques (like multi-dataset training) that might close the gap between machine and human robustness.
  3. Determine if the failure is due to "brittleness" in the models' language representations or specific biases in the SQuAD-style training process.

Summary updated

Abstract

We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data. Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set. Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting. The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts. Across a broad range of models, we observe average performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast, a strong human baseline matches or exceeds the performance of SQuAD models on the original domain and exhibits little to no drop in new domains. Taken together, our results confirm the surprising resilience of the holdout method and emphasize the need to move towards evaluation metrics that incorporate robustness to natural distribution shifts.
BibTeX
@inproceedings{miller2020effect,
  author = {John Miller and Karl Krauth and Benjamin Recht and Ludwig Schmidt},
  title = {{The Effect of Natural Distribution Shift on Question Answering Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=Hh6cn4AvLt},
}
Figure