G-DAUG: Generative Data Augmentation for Commonsense Reasoning

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey

TL;DR

This paper introduces G-DAUG, a novel generative data augmentation framework that creates synthetic examples for commonsense reasoning tasks. By leveraging pretrained language models and employing smart selection strategies, G-DAUG significantly improves model performance, robustness, and generalization, even in low-resource settings, setting new state-of-the-art results on several benchmarks.

What Problem Does This Paper Address?

Modern AI models achieve impressive performance on commonsense reasoning tasks, but this success heavily relies on massive, human-annotated training datasets. Creating these datasets is costly and time-consuming. More importantly, human-authored data can contain subtle "annotation artifacts" that models can exploit, leading to brittle performance that doesn't generalize well to unseen or slightly altered data. This paper tackles the challenge of augmenting existing datasets with high-quality, synthetic examples to improve both in-distribution accuracy and out-of-distribution robustness without the prohibitive cost of manual annotation.

Methods

The core of G-DAUG is a generative approach to data augmentation. Here's a breakdown of their key techniques:

  • Generative Data Augmentation (G-DAUG): The framework uses pretrained language models (like GPT-2) to generate synthetic question-answer pairs. This includes generating questions, answers, and distractors.
  • Data Selection: Simply generating a large pool of synthetic data isn't enough; it can be noisy or redundant. G-DAUG employs two novel selection strategies:
  • Influence Functions: This method identifies and filters out synthetic examples that are likely to increase the model's validation loss, effectively removing detrimental examples.
  • Diversity Maximization: This approach uses a greedy algorithm to select synthetic examples that maximize the diversity of the generated dataset, ensuring a broader coverage of linguistic and semantic phenomena.
  • G-DAUG-Combo: A combined strategy that first filters with influence functions and then selects for diversity.
  • Two-Stage Training: To mitigate potential label noise in synthetic data, G-DAUG uses a two-stage training process: first, train on the selected synthetic data, and then further train on the original, human-authored data. This allows the model to learn from the synthetic data while correcting any learned noise.

Datasets

The authors evaluate G-DAUG on several commonsense reasoning benchmarks:

  • COMMONSENSEQA: A multiple-choice QA dataset for commonsense reasoning.
  • WINOGRANDE: A larger, more challenging version of the Winograd Schema Challenge.
  • CODAH: An adversarially-constructed benchmark for commonsense reasoning.
  • HellaSwag: A dataset for commonsense inference.

They also test its applicability on:

  • ARC-Challenge: A scientific QA dataset.
  • SNLI: A natural language inference dataset.

For low-resource experiments, they downsample HellaSwag and SNLI.

Key Insights and Ideas

  • Pretrained Generators are Crucial: The paper strongly suggests that using pretrained language models for generation is critical. Using a generator trained from scratch yields only marginal improvements, while a pretrained generator leads to significant gains. This highlights the value of the commonsense knowledge already encoded in large pretrained models.
  • Synthetic Labels Matter: The correctness of the generated labels is paramount. Randomly generated labels significantly hurt performance, emphasizing that the quality of the synthetic data's "answers" is as important as the questions themselves.
  • G-DAUG Introduces Novelty: Analysis shows that G-DAUG's generated questions introduce many new semantic units (entities, relations) not present in the original training data, suggesting it expands the model's knowledge base rather than just rephrasing existing information.
  • Diversity and Quality are Complementary: While both influence-based filtering (quality) and diversity maximization individually improve performance, their combination (G-DAUG-Combo) yields the best results, indicating that both aspects are crucial for effective data augmentation.
  • Low-Resource Settings Benefit Most: G-DAUG provides the largest gains in low-resource scenarios, effectively increasing the utility of limited training data.

Key Results

  • State-of-the-Art Performance: G-DAUG-Combo achieved new state-of-the-art results on WINOGRANDE, CODAH, and COMMONSENSEQA.
  • Significant Accuracy Gains: On WINOGRANDE, G-DAUG-Combo provided a 5.0% absolute gain over previous SOTA. On COMMONSENSEQA, it improved by 0.4% over the previous non-ensemble SOTA.
  • Improved Robustness: G-DAUG-trained models showed significantly better robustness against adversarial attacks (TextFooler) and perturbed evaluation sets (synonym replacement) compared to baselines. G-DAUG-Diversity was particularly effective in improving robustness.
  • Effective in Low-Resource Settings: For smaller training set sizes on WINOGRANDE, G-DAUG-Combo effectively quadrupled the training data's impact.
  • Two-Stage Training is Key: The proposed two-stage training method outperformed simpler mixing or importance-weighted loss approaches for mitigating synthetic data noise.

Limitations and Future Work

  • Human Labeling is Still Superior (but costly): While G-DAUG is effective, human-labeled data, even when used sparingly, can achieve comparable performance with significantly less data. The authors note that human labeling is expensive, implying G-DAUG offers a cost-effective alternative.
  • Task-Dependent Labeling Utility: The effectiveness of self-supervised relabeling with a task model was found to be task-dependent, suggesting further research into optimal relabeling strategies.
  • Future Directions: The authors suggest exploring more sophisticated methods for enhancing the quality and diversity of generated data, and investigating human-in-the-loop approaches for relabeling. They also plan to explore G-DAUG's applicability to other NLP tasks beyond commonsense reasoning.

Summary updated

Abstract

Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance. However, manual curation of training examples is expensive and has been shown to introduce annotation artifacts that neural models can readily exploit and overfit on. We investigate G-DAUG^C, a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. Our approach generates synthetic examples using pretrained language models, and selects the most informative and diverse set of examples for data augmentation. In experiments with multiple commonsense reasoning benchmarks, G-DAUG^C consistently outperforms existing data augmentation methods based on back-translation, and establishes a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA. Further, in addition to improvements in in-distribution accuracy, G-DAUG^C-augmented training also enhances out-of-distribution generalization, showing greater robustness against adversarial or perturbed examples. Our analysis demonstrates that G-DAUG^C produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance. Our findings encourage future research toward generative data augmentation to enhance both in-distribution learning and out-of-distribution generalization.
BibTeX
@inproceedings{yang2020g,
  author = {Yiben Yang and Chaitanya Malaviya and Jared Fernandez and Swabha Swayamdipta and Ronan Le Bras and Ji-Ping Wang and Chandra Bhagavatula and Yejin Choi and Doug Downey},
  title = {{G-DAUG: Generative Data Augmentation for Commonsense Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=S9UZ_znEGm5},
}
Figure