G-DAug: Generative Data Augmentation for Commonsense Reasoning

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey

Abstract

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction: The Commonsense Bottleneck

Teaching machines "common sense" is notoriously difficult because humans rarely state the obvious. We don't write down that "gravity pulls things down" or "people get wet in the rain"—we just assume it. Consequently, AI models require massive, human-annotated datasets to learn these implicit rules.

However, human-authored data has two major flaws:

  1. Expense: Crowdsourcing tens of thousands of nuanced questions is slow and costly.
  2. Annotation Artifacts: Humans are predictable. We often leave unintentional "shortcuts" in data (e.g., using certain words more often in correct answers) that models exploit to get high scores without actually "reasoning."

The authors ask: Can we use the implicit knowledge already buried in pretrained language models to generate a nearly infinite supply of diverse, high-quality training data?

The G-DAUG Framework

The authors introduce G-DAUG$^c$ (Generative Data Augmentation for Commonsense), a multi-stage pipeline designed to scale training data without sacrificing quality.

The G-DAUG framework pipeline: (1) Synthetic generation, (2) Relabeling, (3) Influence filtering, (4) Diversity filtering, (5) Synthetic training, and (6) Organic training.

The pipeline consists of five key stages:

  1. Synthetic Generation: A generator (GPT-2) is fine-tuned on a small set of human questions to produce new questions, correct answers, and "distractors" (wrong choices).
  2. Data Relabeling: To ensure label accuracy, a "task model" (RoBERTa) predicts the correct answer for the newly generated questions.
  3. The "Influence" Filter: This core innovation uses Influence Functions to estimate which synthetic examples would actually improve a model's performance on a validation set. Detrimental examples are discarded.
  4. Diversity Selection: To avoid redundancy, a greedy algorithm selects a subset of examples that maximizes unique unigrams, ensuring a wide variety of concepts.
  5. Two-Stage Training: The model is first trained on the synthetic data (Synthetic Training) and then "fine-tuned" on the original human data (Organic Training). This allows the model to learn from the volume of synthetic data while correcting for noise.

Datasets and Evaluation

The framework was evaluated across several prominent commonsense and logic benchmarks, including WINOGRANDE (pronoun resolution), CODAH (adversarial common sense), COMMONSENSEQA (external knowledge), and HellaSwag (story continuation). It was also tested on SNLI and ARC-Challenge to demonstrate extensibility to natural language inference and science questions.

Key Insights

  • LMs as Implicit Knowledge Bases: Models like GPT-2 capture significant world knowledge during pretraining, which can be extracted to train other models.
  • Quality Over Quantity: Simply adding more synthetic data is insufficient. Filtering via Influence Functions is critical to prevent "noise" from hurting performance.
  • Diversity and Robustness: Selecting for diversity (G-DAUG-Diversity) significantly enhances model resistance to adversarial "word-swap" attacks.
  • The "Organic" Finish: The two-stage training approach treats synthetic data as a "warm-up" and human data as the "ground truth," preventing overfitting to generator quirks.

Key Results

G-DAUG$^c$ established new state-of-the-art (SOTA) scores on WINOGRANDE (a 5.0% absolute gain over previous non-ensemble methods), CODAH, and COMMONSENSEQA.

Accuracy vs. Training Set Size: G-DAUG-Combo consistently outperforms RoBERTa and Backtranslation, especially in low-resource settings.
  • Low-Resource Power: In settings where human data is scarce, G-DAUG$^c$ effectively quadrupled the training signal. Training on a tiny "Extra Small" dataset with G-DAUG matched the performance of a model trained on a "Medium" human dataset.
  • Adversarial Robustness: Models trained with G-DAUG were harder to "fool." When attacked by TextFooler, G-DAUG models maintained higher accuracy and required the attacker to change more words to succeed.

Limitations

While powerful, G-DAUG faces challenges:

  • Generative Noise: Approximately 22% of generated questions were flagged by humans as ambiguous.
  • Computational Cost: Calculating influence functions is expensive, though the authors provide scalable stochastic estimation methods.
  • Future Work: Integrating "active learning" could allow humans to label only the most informative synthetic examples identified by influence functions.

Summary updated

BibTeX
@inproceedings{yang2020g,
  author = {Yiben Yang and Chaitanya Malaviya and Jared Fernandez and Swabha Swayamdipta and Ronan Le Bras and Ji-Ping Wang and Chandra Bhagavatula and Yejin Choi and Doug Downey},
  title = {{G-DAug: Generative Data Augmentation for Commonsense Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=S9UZ_znEGm5},
}
Figure