Synthetic QA Corpora Generation with Roundtrip Consistency

Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, Michael Collins

Abstract: Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, Michael Collins. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

Large-scale language models like BERT have significantly advanced Question Answering (QA), yet they remain constrained by the availability of high-quality labeled data. While unlabeled text is abundant, human-annotated QA pairs are expensive and scarce. Standard pretraining tasks, such as masked language modeling, do not explicitly teach models the mechanics of QA—specifically, how questions relate to precise spans of text.

To bridge this gap, researchers from Google Research propose a method to generate millions of high-quality synthetic (Context, Question, Answer) triples from unlabeled text. The core challenge in synthetic generation is noise; models often produce nonsensical questions or pairs where the question does not lead back to the intended answer. The authors address this through a "roundtrip consistency" filter.

The Roundtrip Consistency Pipeline

The innovation lies in a four-step pipeline designed to ensure that generated questions are reliable and answerable.

The four-step roundtrip consistency pipeline for generating synthetic QA pairs.
  1. Answer Extraction ($C \rightarrow A$): A model identifies a candidate answer span ($A$) within a passage ($C$).
  2. Question Generation ($C, A \rightarrow Q$): A second model generates a question ($Q$) that should be answered by $A$ in context $C$.
  3. Answer Prediction ($C, Q \rightarrow A'$): A standard QA model attempts to answer the generated question $Q$ using context $C$.
  4. Roundtrip Consistency Check ($A \stackrel{?}{=} A'$): The triple is kept only if the predicted answer ($A'$) matches the original answer ($A$).

This filtering mechanism is crucial. The authors found that among discarded triples, only 16% were correct, whereas 39% of those passing the consistency check were accurate. This ensures the pretraining signal is not corrupted by model hallucinations.

Scaling and Filtering Impact

The researchers evaluated their approach on two major benchmarks: SQuAD 2.0 and Natural Questions (NQ). They used the full text from the NQ training set (approx. 1 billion words) as the source for unlabeled text.

Performance on SQuAD 2.0 dev set scales with the number of synthetic examples and improves significantly with roundtrip consistency (RT) filtering.

As shown in the figure above, performance scales consistently with the volume of synthetic data. Crucially, models pretrained on data filtered for roundtrip consistency (RT) consistently outperformed those trained on unfiltered synthetic data ("no-RT"). This confirms that quality control via self-consistency is a powerful substitute for human annotation.

Key Results and Insights

The results demonstrate that synthetic data acts as an effective intermediate pretraining step:

  • SQuAD 2.0: An ensemble model achieved 86.0 Exact Match (EM), coming within 0.1% of human performance (86.8 EM).
  • Natural Questions: The method established a new state-of-the-art, reducing the gap between model and human performance on short answers by 50% for a single model.
  • Diversity: Combining different styles of synthetic data (e.g., NQ-style and SQuAD-style) improved robustness, suggesting that exposure to diverse question formats is beneficial.

Limitations and Future Work

While highly effective, the approach is computationally intensive, requiring the generation and pretraining of up to 50 million triples. Furthermore, the generator models themselves require an initial seed of labeled data, meaning the process is semi-supervised rather than fully unsupervised. Future research may explore applying these techniques to cross-lingual QA or more complex reasoning tasks where labeled data is even more limited.

Summary updated

BibTeX
@inproceedings{alberti2019synthetic,
  author = {Chris Alberti and Daniel Andor and Emily Pitler and Jacob Devlin and Michael Collins},
  title = {{Synthetic QA Corpora Generation with Roundtrip Consistency}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=uwBwqk-g3-r},
}
Figure