Introduction
Modern Question Answering (QA) models demonstrate impressive performance on standard benchmarks like SQuAD. However, they remain remarkably fragile, often relying on "spurious statistical patterns"—shortcuts based on word overlaps—rather than genuine linguistic comprehension. To address this, researchers have turned to Dynamic Adversarial Data Collection, where humans are tasked with writing questions specifically designed to exploit model weaknesses. While effective, this process is prohibitively expensive and slow, hindering the generation of the massive datasets required for deep learning. This paper explores whether a small seed of human-written adversarial data can be used to train a machine to generate high-quality adversarial examples at scale.
The Synthetic Data Generation Pipeline
The researchers developed a four-stage pipeline designed to "amplify" human creativity into a massive synthetic dataset.
- Passage Selection: Source texts are retrieved from Wikipedia to provide a diverse range of contexts.
- Answer Candidate Selection: Moving beyond simple noun-phrase extraction, the authors introduced a Self-Attention Labelling (SAL) head. This component identifies spans within a text that are likely to serve as answers to challenging questions, outperforming traditional span-extraction methods.
- Question Generation: A BART-Large model is fine-tuned on a combination of SQuAD and human-authored adversarial questions. This training enables the generator to produce questions requiring complex reasoning rather than simple keyword matching.
- Filtering and Re-labelling: To ensure data quality, an ensemble of six QA models evaluates the generated pairs. If the models disagree, the example is either discarded or re-labelled based on the ensemble's consensus, a process known as self-training.
Experimental Setup and Datasets
The methodology was evaluated against several rigorous benchmarks:
- SQuAD 1.1: A baseline for standard extractive QA.
- AdversarialQA: Three subsets (DBiDAF, DBERT, DRoBERTa) where humans previously attempted to fool specific model architectures.
- MRQA: A suite of 12 datasets used to test out-of-distribution generalization across various domains, such as medical and news texts.
- CheckList: A set of linguistic probes evaluating specific skills like negation, coreference, and taxonomic reasoning.
Key Results and Performance Analysis
The proposed "SynQA" approach yielded significant improvements in model robustness and generalization. The model achieved a new state-of-the-art on the AdversarialQA dataset, improving performance by 3.7 F1 points.
Crucially, the robustness was validated through live human-in-the-loop testing. When crowdworkers were incentivized to fool the models, they succeeded 17.6% of the time against a standard baseline but only 8.8% of the time against the SynQA-trained model. Furthermore, the model showed enhanced performance on 9 out of 12 MRQA datasets, indicating that training on synthetic adversarial data helps the model learn more robust, generalizable language features.
Discussion and Future Work
The core innovation of this work lies in amplification. By using human adversarial examples as a seed, the synthetic generator inherits the complexity and "hardness" of human-authored questions. The filtering process, based on ensemble agreement rather than simple confidence scores, proved vital in maintaining dataset quality.
Despite these gains, limitations remain. Synthetic generators occasionally produce questions requiring external knowledge not present in the text or exhibit minor grammatical inconsistencies. Future research will focus on increasing the diversity of synthetic questions and exploring decoding strategies that encourage greater creative variation in the generated adversarial examples.
Summary updated