Abstract
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction: The Commonsense Bottleneck
Teaching machines "common sense" is notoriously difficult because humans rarely state the obvious. We don't write down that "gravity pulls things down" or "people get wet in the rain"—we just assume it. Consequently, AI models require massive, human-annotated datasets to learn these implicit rules.
However, human-authored data has two major flaws:
- Expense: Crowdsourcing tens of thousands of nuanced questions is slow and costly.
- Annotation Artifacts: Humans are predictable. We often leave unintentional "shortcuts" in data (e.g., using certain words more often in correct answers) that models exploit to get high scores without actually "reasoning."
The authors ask: Can we use the implicit knowledge already buried in pretrained language models to generate a nearly infinite supply of diverse, high-quality training data?
The G-DAUG Framework
The authors introduce G-DAUG$^c$ (Generative Data Augmentation for Commonsense), a multi-stage pipeline designed to scale training data without sacrificing quality.
The pipeline consists of five key stages:
- Synthetic Generation: A generator (GPT-2) is fine-tuned on a small set of human questions to produce new questions, correct answers, and "distractors" (wrong choices).
- Data Relabeling: To ensure label accuracy, a "task model" (RoBERTa) predicts the correct answer for the newly generated questions.
- The "Influence" Filter: This core innovation uses Influence Functions to estimate which synthetic examples would actually improve a model's performance on a validation set. Detrimental examples are discarded.
- Diversity Selection: To avoid redundancy, a greedy algorithm selects a subset of examples that maximizes unique unigrams, ensuring a wide variety of concepts.
- Two-Stage Training: The model is first trained on the synthetic data (Synthetic Training) and then "fine-tuned" on the original human data (Organic Training). This allows the model to learn from the volume of synthetic data while correcting for noise.
Datasets and Evaluation
The framework was evaluated across several prominent commonsense and logic benchmarks, including WINOGRANDE (pronoun resolution), CODAH (adversarial common sense), COMMONSENSEQA (external knowledge), and HellaSwag (story continuation). It was also tested on SNLI and ARC-Challenge to demonstrate extensibility to natural language inference and science questions.
Key Insights
- LMs as Implicit Knowledge Bases: Models like GPT-2 capture significant world knowledge during pretraining, which can be extracted to train other models.
- Quality Over Quantity: Simply adding more synthetic data is insufficient. Filtering via Influence Functions is critical to prevent "noise" from hurting performance.
- Diversity and Robustness: Selecting for diversity (G-DAUG-Diversity) significantly enhances model resistance to adversarial "word-swap" attacks.
- The "Organic" Finish: The two-stage training approach treats synthetic data as a "warm-up" and human data as the "ground truth," preventing overfitting to generator quirks.
Key Results
G-DAUG$^c$ established new state-of-the-art (SOTA) scores on WINOGRANDE (a 5.0% absolute gain over previous non-ensemble methods), CODAH, and COMMONSENSEQA.
- Low-Resource Power: In settings where human data is scarce, G-DAUG$^c$ effectively quadrupled the training signal. Training on a tiny "Extra Small" dataset with G-DAUG matched the performance of a model trained on a "Medium" human dataset.
- Adversarial Robustness: Models trained with G-DAUG were harder to "fool." When attacked by TextFooler, G-DAUG models maintained higher accuracy and required the attacker to change more words to succeed.
Limitations
While powerful, G-DAUG faces challenges:
- Generative Noise: Approximately 22% of generated questions were flagged by humans as ambiguous.
- Computational Cost: Calculating influence functions is expensive, though the authors provide scalable stochastic estimation methods.
- Future Work: Integrating "active learning" could allow humans to label only the most informative synthetic examples identified by influence functions.
Summary updated