This paper presents a large-scale randomized controlled study investigating whether Adversarial Data Collection (ADC)—a popular "model-in-the-loop" approach to training AI—actually makes models more robust. While ADC helps models perform better on other adversarial benchmarks, the authors find it often degrades performance on diverse, "natural" out-of-domain datasets compared to standard data collection methods.
TL;DR
Adversarial data collection (ADC) creates "harder" datasets by having humans try to fool models in real-time, but this doesn't necessarily lead to better general-purpose AI. This study shows that while ADC-trained models excel at beating other adversarial tests, they often lose their edge on diverse, real-world datasets because the collection process skews the data distribution toward specific "tricks."
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) models are notorious for being "brittle." They often succeed on benchmarks by exploiting spurious associations—shortcuts or patterns in the data that don't represent true reasoning (e.g., always associating the word "not" with a negative sentiment).
To fix this, researchers turned to ADC: a human worker interacts with a model and gets a bonus for writing questions the model gets wrong. The hope is that by training on these "failures," models will learn more robust features. However, it’s been unclear if this process actually produces better generalizers or if it just teaches models to avoid a new set of specific adversarial traps.
Methods
The authors conducted a rigorous Randomized Controlled Trial (RCT) for data collection. They recruited crowdworkers and randomly assigned them to one of three settings:
- ADC-BERT: Workers tried to fool a fine-tuned BERT-large model.
- ADC-ELECTRA: Workers tried to fool a fine-tuned ELECTRA-large model.
- Standard (SDC): Workers wrote questions without any model feedback.
By keeping the source passages identical across groups and varying only the "model-in-the-loop" incentive, the authors could isolate the impact of the adversarial process itself. They then fine-tuned BERT, RoBERTa, and ELECTRA models on these different datasets and evaluated them across a massive battery of tests.
Datasets
- Source Data: Context passages were the first 100 words of Wikipedia articles from a variant of the Natural Questions (NQ) dataset.
- New Benchmarks: The study produced several new datasets (BERT-fooled, ELECTRA-fooled, etc.) totaling over 150k question-answer pairs before quality filtering.
- Evaluation Sets: Models were tested on their own held-out data, adversarial sets from prior work (Bartolo et al., 2020), and 12 MRQA datasets (including SQuAD, HotpotQA, and TriviaQA) to measure out-of-domain generalization.
Key Insights and Ideas
- The "Intent" to Fool Matters: Interestingly, models trained on any data from the ADC process (even questions that failed to fool the model) performed similarly to those trained only on successful "fooled" examples. This suggests the adversarial mindset of the worker changes the data more than the model's specific failures do.
- Distribution Shift: ADC significantly skews what kind of questions get written. When workers see a model is good at "Who" or "When" questions, they stop writing them and pivot to harder types, like numerical reasoning.
- Robustness vs. Specialization: ADC makes models better at handling adversarial attacks, but it can make them worse at "natural" language. It’s a trade-off: you gain resistance to specific hacks but lose performance on the broad distribution of human language.

Key Results
- Adversarial Success: Models trained on ADC data consistently outperformed SDC-trained models on adversarial test sets. For example, a RoBERTa model trained on BERT-fooled data beat the SDC-trained version by over 6 EM (Exact Match) points on adversarial benchmarks.
- Generalization Failure: On the 12 MRQA datasets, the story flipped. For BERT, fine-tuning on standard (SDC) data led to significantly better performance on 9 out of 12 datasets compared to ADC data.
- Numerical Reasoning: Qualitative analysis showed that ADC datasets contained much higher proportions of questions requiring numerical reasoning (e.g., "How many..."), likely because workers discovered this was a common model weakness.
Limitations and Future Work
The authors acknowledge that ADC is significantly more expensive and time-consuming than standard collection due to the need for real-time model hosting and higher worker bonuses.
They suggest that instead of asking workers to write questions from scratch, a better path forward might be adversarial editing: giving workers existing "natural" questions and asking them to modify them until the model fails. This could potentially combine the best of both worlds—maintaining a natural data distribution while still forcing the model to confront its weaknesses. Future research will also need to explore how factors like passage length and domain complexity interact with the ADC process.
Summary updated