Unsupervised Adaptation of Question Answering Systems via Generative Self-training

Steven J. Rennie, Etienne Marcheret, Neil Mallinar, David Nahamoo, Vaibhava GoelEMNLP 2020 · OpenReview

Abstract

This paper introduces a generative self-training framework designed to adapt Question Answering (QA) systems to new, unlabeled domains. By iteratively generating synthetic QA pairs and filtering them using a theoretically grounded "Question Answer Posterior" (QAP) metric, the authors achieve state-of-the-art adaptation performance. The method is significantly more efficient than previous approaches, requiring an order of magnitude less data and computation while providing a formal theoretical foundation for common heuristics like Roundtrip Consistency.

Introduction

While BERT-style Transformer models have revolutionized QA, they typically require massive amounts of human-labeled data to perform well on specific domains. When moved to a new "target" domain (e.g., from Wikipedia to technical manuals), performance often drops. Existing solutions usually involve "pre-training" on millions of synthetic QA pairs, which is computationally expensive and often relies on Roundtrip Consistency (RTC)—a heuristic that lacked a solid theoretical foundation. This paper addresses these limitations through a principled iterative adaptation approach.

Methodology

The authors propose an Iterative Pre-train then Fine-tune approach. The process works in a continuous loop:

  1. Generate: Use current models to create synthetic QA pairs from unlabeled target text.
  2. Filter/Weight: Use the novel Question Answer Posterior (QAP) filtering mechanism.
  3. Pre-train: Train the QA system from scratch on this filtered synthetic data.
  4. Fine-tune: Finalize the model on available ground-truth data.
  5. Repeat: Use the improved model to generate higher-quality synthetic data in the next iteration.
Overview of the generative self-training framework: Pre-training on synthetic QA pairs followed by fine-tuning on ground-truth data.

The QAP Framework

The key innovation is the QAP framework, which provides a theoretical link for why Roundtrip Consistency works. The authors prove that RTC is a specific instance of maximizing a lower bound on the probability of observed text. They generalize this into "soft" filtering (QAP), where synthetic pairs are weighted by the probability $p(A|Q, C)$ rather than being discarded by a binary pass/fail. This allows the model to learn from "mostly correct" samples, enhancing training stability and performance.

Comparison of QAP selection and weighting strategies against standard heuristics like Roundtrip Consistency (RTC).

Experimental Setup

The researchers evaluated their approach across three scenarios:

  • SQuAD 1.1: Used for "Well-Matched" conditions.
  • Natural Questions (NQ): Used for "Mismatched" conditions, featuring longer, more abstractive answers.
  • Unlabeled Contexts: Adaptation was performed on the development sets of these corpora, treated as unlabeled data.

Key Results and Insights

  • Superior Performance: On SQuAD 1.1, the QAP-adapted model achieved an F1 score of 81.8, significantly outperforming the baseline (79.7).
  • Efficiency through Pruning: By discarding low-quality synthetic data early, the method achieves better results with 10x less data and significantly faster training times (e.g., 6 hours vs. 26 hours for the first iteration).
  • The Power of Iteration: As the QA system improves at answering, the question generator becomes more effective at producing high-quality training samples, creating a beneficial feedback loop.

Conclusion

This work demonstrates that task-specific generative training is more effective for QA adaptation than general language modeling. While challenges remain in handling specific structural biases like answer length in NQ, the QAP framework offers a robust, theoretically sound, and computationally efficient path forward for domain-specific QA systems.

Summary updated

BibTeX
@inproceedings{rennie2020unsupervised,
  author = {Steven J. Rennie and Etienne Marcheret and Neil Mallinar and David Nahamoo and Vaibhava Goel},
  title = {{Unsupervised Adaptation of Question Answering Systems via Generative Self-training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=fepouk9EDU1},
}
Figure