On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study

Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih

This paper presents a large-scale randomized controlled study investigating whether Adversarial Data Collection (ADC)—a popular "model-in-the-loop" approach to training AI—actually makes models more robust. While ADC helps models perform better on other adversarial benchmarks, the authors find it often degrades performance on diverse, "natural" out-of-domain datasets compared to standard data collection methods.

TL;DR

Adversarial data collection (ADC) creates "harder" datasets by having humans try to fool models in real-time, but this doesn't necessarily lead to better general-purpose AI. This study shows that while ADC-trained models excel at beating other adversarial tests, they often lose their edge on diverse, real-world datasets because the collection process skews the data distribution toward specific "tricks."

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) models are notorious for being "brittle." They often succeed on benchmarks by exploiting spurious associations—shortcuts or patterns in the data that don't represent true reasoning (e.g., always associating the word "not" with a negative sentiment).

To fix this, researchers turned to ADC: a human worker interacts with a model and gets a bonus for writing questions the model gets wrong. The hope is that by training on these "failures," models will learn more robust features. However, it’s been unclear if this process actually produces better generalizers or if it just teaches models to avoid a new set of specific adversarial traps.

Methods

The authors conducted a rigorous Randomized Controlled Trial (RCT) for data collection. They recruited crowdworkers and randomly assigned them to one of three settings:

  1. ADC-BERT: Workers tried to fool a fine-tuned BERT-large model.
  2. ADC-ELECTRA: Workers tried to fool a fine-tuned ELECTRA-large model.
  3. Standard (SDC): Workers wrote questions without any model feedback.
The worker interface for the ADC setting, where they receive real-time feedback on whether they successfully fooled the model.

By keeping the source passages identical across groups and varying only the "model-in-the-loop" incentive, the authors could isolate the impact of the adversarial process itself. They then fine-tuned BERT, RoBERTa, and ELECTRA models on these different datasets and evaluated them across a massive battery of tests.

Datasets

  • Source Data: Context passages were the first 100 words of Wikipedia articles from a variant of the Natural Questions (NQ) dataset.
  • New Benchmarks: The study produced several new datasets (BERT-fooled, ELECTRA-fooled, etc.) totaling over 150k question-answer pairs before quality filtering.
  • Evaluation Sets: Models were tested on their own held-out data, adversarial sets from prior work (Bartolo et al., 2020), and 12 MRQA datasets (including SQuAD, HotpotQA, and TriviaQA) to measure out-of-domain generalization.

Key Insights and Ideas

  • The "Intent" to Fool Matters: Interestingly, models trained on any data from the ADC process (even questions that failed to fool the model) performed similarly to those trained only on successful "fooled" examples. This suggests the adversarial mindset of the worker changes the data more than the model's specific failures do.
  • Distribution Shift: ADC significantly skews what kind of questions get written. When workers see a model is good at "Who" or "When" questions, they stop writing them and pivot to harder types, like numerical reasoning.
  • Robustness vs. Specialization: ADC makes models better at handling adversarial attacks, but it can make them worse at "natural" language. It’s a trade-off: you gain resistance to specific hacks but lose performance on the broad distribution of human language.

Comparison of wh-question frequency in the BERT-fooled ADC set. Notice the high proportion of "what" questions. The distribution in the standard (SDC) set is more balanced, with significantly more "who" and "when" questions.

Key Results

  • Adversarial Success: Models trained on ADC data consistently outperformed SDC-trained models on adversarial test sets. For example, a RoBERTa model trained on BERT-fooled data beat the SDC-trained version by over 6 EM (Exact Match) points on adversarial benchmarks.
  • Generalization Failure: On the 12 MRQA datasets, the story flipped. For BERT, fine-tuning on standard (SDC) data led to significantly better performance on 9 out of 12 datasets compared to ADC data.
  • Numerical Reasoning: Qualitative analysis showed that ADC datasets contained much higher proportions of questions requiring numerical reasoning (e.g., "How many..."), likely because workers discovered this was a common model weakness.

Limitations and Future Work

The authors acknowledge that ADC is significantly more expensive and time-consuming than standard collection due to the need for real-time model hosting and higher worker bonuses.

They suggest that instead of asking workers to write questions from scratch, a better path forward might be adversarial editing: giving workers existing "natural" questions and asking them to modify them until the model fails. This could potentially combine the best of both worlds—maintaining a natural data distribution while still forcing the model to confront its weaknesses. Future research will also need to explore how factors like passage length and domain complexity interact with the ADC process.

Summary updated

Abstract

In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that models trained on these more challenging datasets will rely less on superficial patterns, and thus be less brittle. However, despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models. In this paper, we conduct a large-scale controlled study focused on question answering, assigning workers at random to compose questions either (i) adversarially (with a model in the loop); or (ii) in the standard fashion (without a model). Across a variety of models and datasets, we find that models trained on adversarial data usually perform better on other adversarial datasets but worse on a diverse collection of out-of-domain evaluation sets. Finally, we provide a qualitative analysis of adversarial (vs standard) data, identifying key differences and offering guidance for future research.
BibTeX
@inproceedings{kaushik2021efficacy,
  author = {Divyansh Kaushik and Douwe Kiela and Zachary C. Lipton and Wen-tau Yih},
  title = {{On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=YoNCjDoK2mn},
}
Figure