TL;DR
This paper introduces a novel method for generating synthetic question-answering (QA) data by combining question generation and answer extraction models, then filtering for "roundtrip consistency." Pretraining on this synthetic data significantly boosts performance on SQuAD2 and NQ, achieving state-of-the-art on NQ.
What Problem Does This Paper Address?
The success of large language models in QA has largely been driven by pretraining on massive unlabeled text corpora and then fine-tuning on smaller, human-annotated QA datasets. However, creating these high-quality, human-annotated datasets is expensive and time-consuming. This paper addresses the challenge of leveraging the vast amount of readily available unlabeled text to create synthetic QA data that can augment or even replace some of the need for human annotation, thereby improving QA model performance, especially in low-resource scenarios.
Methods
The core of the paper is a three-step process for generating synthetic QA triples (Context, Question, Answer):
- Answer Extraction: A model predicts a short, extractive answer (A) from a given context (C).
- Question Generation: A model generates a question (Q) conditioned on the context (C) and the extracted answer (A).
- Answer Re-extraction: A model attempts to extract an answer (A') from the context (C) given the generated question (Q).
The key innovation is roundtrip consistency: a synthetic triple (C, Q, A) is kept only if the re-extracted answer (A') matches the original extracted answer (A). This filtering step is crucial for ensuring the quality and coherence of the generated data. The authors explore two main approaches:
- Fine-tuning Only: They fine-tune publicly available BERT models for each of the three steps.
- Full Pretraining: For question generation, they employ a more powerful sequence-to-sequence Transformer model that undergoes full pretraining.
Datasets
- Training/Pretraining:
- SQuAD2 (Rajpurkar et al., 2018)
- NQ (Natural Questions) (Kwiatkowski et al., 2019)
- Wikipedia text (used for generating synthetic data)
- Evaluation:
- SQuAD2
- NQ
Key Insights and Ideas
- Roundtrip Consistency is Key: The paper strongly emphasizes that filtering generated QA pairs for roundtrip consistency (i.e., the generated question, when used to find an answer in the original context, yields the same answer as initially extracted) is critical for producing high-quality synthetic data. This acts as a powerful self-supervision signal.
- Synthetic Data for Pretraining Works: Pretraining QA models on large amounts of synthetically generated data, even when derived from models that are only fine-tuned, leads to significant performance improvements on downstream QA tasks.
- Leveraging Unlabeled Text is Powerful: The ability to generate millions of synthetic QA examples from readily available unlabeled text (like Wikipedia) offers a scalable way to improve QA systems, potentially reducing reliance on expensive human annotation.
- Different Datasets, Different Styles: The paper highlights that SQuAD2 and NQ generate question-answer pairs with distinct styles. SQuAD2 models tend to focus on specific entity properties, while NQ models create questions around broader themes.
Key Results
- State-of-the-Art on NQ: The proposed method, particularly the full pretraining approach, achieved a new state-of-the-art on the Natural Questions (NQ) dataset.
- Significant Improvements on SQuAD2: Pretraining on synthetic data led to substantial gains on SQuAD2, with the full pretraining approach achieving exact match (EM) and F1 scores very close to human performance (0.1-0.4% difference).
- Roundtrip Filtering Benefits: Experiments (e.g., Figure 1) show that models pretrained on roundtrip-consistent synthetic data consistently outperform those trained without this filtering step.
- Data Diversity Matters: Pretraining on a mix of SQuAD2 and NQ synthetic data was beneficial for SQuAD2 performance, suggesting that diverse synthetic data sources can be advantageous.
Limitations and Future Work
- Formal Grounding: The authors acknowledge that a formal theoretical justification for why roundtrip consistency works is still an area for future research, though they sketch a potential approach inspired by prior work on learning with labeled and unlabeled data.
- Combining Approaches: They suggest that combining their generative approach with methods that model the joint distribution of questions and answers could be a fruitful direction for future work.
- Further Exploration of Auxiliary Functions: The paper proposes auxiliary functions for formal grounding and suggests further investigation into their connection to the presented algorithms.
- Human Performance Gap: While the results are impressive, there remains a small gap between the best model performance and human performance on SQuAD2.
Summary updated