Abstract
This paper introduces a generative self-training framework designed to adapt Question Answering (QA) systems to new, unlabeled domains. By iteratively generating synthetic QA pairs and filtering them using a theoretically grounded "Question Answer Posterior" (QAP) metric, the authors achieve state-of-the-art adaptation performance. The method is significantly more efficient than previous approaches, requiring an order of magnitude less data and computation while providing a formal theoretical foundation for common heuristics like Roundtrip Consistency.
Introduction
While BERT-style Transformer models have revolutionized QA, they typically require massive amounts of human-labeled data to perform well on specific domains. When moved to a new "target" domain (e.g., from Wikipedia to technical manuals), performance often drops. Existing solutions usually involve "pre-training" on millions of synthetic QA pairs, which is computationally expensive and often relies on Roundtrip Consistency (RTC)—a heuristic that lacked a solid theoretical foundation. This paper addresses these limitations through a principled iterative adaptation approach.
Methodology
The authors propose an Iterative Pre-train then Fine-tune approach. The process works in a continuous loop:
- Generate: Use current models to create synthetic QA pairs from unlabeled target text.
- Filter/Weight: Use the novel Question Answer Posterior (QAP) filtering mechanism.
- Pre-train: Train the QA system from scratch on this filtered synthetic data.
- Fine-tune: Finalize the model on available ground-truth data.
- Repeat: Use the improved model to generate higher-quality synthetic data in the next iteration.
The QAP Framework
The key innovation is the QAP framework, which provides a theoretical link for why Roundtrip Consistency works. The authors prove that RTC is a specific instance of maximizing a lower bound on the probability of observed text. They generalize this into "soft" filtering (QAP), where synthetic pairs are weighted by the probability $p(A|Q, C)$ rather than being discarded by a binary pass/fail. This allows the model to learn from "mostly correct" samples, enhancing training stability and performance.
Experimental Setup
The researchers evaluated their approach across three scenarios:
- SQuAD 1.1: Used for "Well-Matched" conditions.
- Natural Questions (NQ): Used for "Mismatched" conditions, featuring longer, more abstractive answers.
- Unlabeled Contexts: Adaptation was performed on the development sets of these corpora, treated as unlabeled data.
Key Results and Insights
- Superior Performance: On SQuAD 1.1, the QAP-adapted model achieved an F1 score of 81.8, significantly outperforming the baseline (79.7).
- Efficiency through Pruning: By discarding low-quality synthetic data early, the method achieves better results with 10x less data and significantly faster training times (e.g., 6 hours vs. 26 hours for the first iteration).
- The Power of Iteration: As the QA system improves at answering, the question generator becomes more effective at producing high-quality training samples, creating a beneficial feedback loop.
Conclusion
This work demonstrates that task-specific generative training is more effective for QA adaptation than general language modeling. While challenges remain in handling specific structural biases like answer length in NQ, the QAP framework offers a robust, theoretically sound, and computationally efficient path forward for domain-specific QA systems.
Summary updated