PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

Patrick S. H. Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, Sebastian RiedelTrans. Assoc. Comput. Linguistics 2021 · DOI · OpenReview

TL;DR

This paper introduces PAQ, a massive dataset of 65 million automatically generated question-answer pairs, designed to improve open-domain question answering (ODQA) systems. They also present RePAQ, a novel QA-pair retriever that leverages PAQ to achieve state-of-the-art accuracy while being significantly faster and more memory-efficient than traditional retrieve-and-read models.

What Problem Does This Paper Address?

Current open-domain question answering (ODQA) systems fall into two main categories: retrieve-and-read models that process large text corpora, and closed-book QA (CBQA) models or QA-pair retrievers that rely on memorized or explicitly retrieved QA pairs. While retrieve-and-read models are accurate, they are computationally expensive. CBQA and QA-pair retrievers are faster and more memory-efficient, offering interpretability, but they suffer from lower accuracy because their training data covers a much smaller knowledge base than comprehensive text corpora like Wikipedia. This paper aims to bridge this accuracy gap by creating a much larger and more comprehensive QA-pair resource.

Methods

The authors introduce a pipeline to generate the PAQ dataset, which involves:

  1. Passage Selection: Identifying Wikipedia passages likely to contain information humans would ask about.
  2. Answer Extraction: Identifying potential answer spans within selected passages.
  3. Question Generation: Generating natural language questions for each answer span.
  4. Global Filtering: A crucial step using an ODQA model to ensure generated questions are consistent and answerable, significantly reducing ambiguity and errors compared to previous methods.

For the QA system, they propose RePAQ, a retrieval model that uses a dense Maximum Inner Product Search (MIPS) retriever. RePAQ embeds questions and QA pairs into a shared vector space for efficient matching. They also introduce a reranker to further improve accuracy by re-evaluating the top retrieved QA pairs. RePAQ is designed to be highly configurable for size, speed, or accuracy.

Datasets

  • PAQ (Probably Asked Questions): A newly introduced dataset of 65 million automatically generated natural language question-answer pairs. It's derived from Wikipedia and aims to cover a much broader range of knowledge than typical QA datasets.
  • Natural Questions (NQ): Used for training and evaluation.
  • TriviaQA: Used for training and evaluation.

Key Insights and Ideas

  • The Power of Scale for QA Pairs: Simply scaling up the number of QA pairs (to 65 million) can dramatically improve the performance of QA-pair based models, bringing them closer to the accuracy of much larger retrieve-and-read systems.
  • Global Filtering is Crucial: The authors emphasize that their "global filtering" step, which uses an ODQA model to validate generated questions against the entire knowledge source, is critical for producing high-quality, consistent QA pairs and achieving strong downstream performance.
  • Retrieval Outperforms Memorization (for this scale): While CBQA models trained on PAQ show improvement, they still lag significantly behind RePAQ. This suggests that for large knowledge bases, explicit retrieval is more effective than trying to memorize everything in model parameters.
  • Selective QA Enables Smart Backoff: RePAQ's confidence scores are well-correlated with accuracy, allowing it to perform "selective QA" by abstaining on uncertain questions. This enables a hybrid system where RePAQ handles most queries quickly, and a more powerful but slower model (like FiD) is called upon only when RePAQ's confidence is low, leading to both speed and accuracy gains.

Key Results

  • PAQ Dataset: Contains 65 million filtered QA pairs, achieving over 90% answer coverage on NQ and TriviaQA validation sets.
  • RePAQ Performance: RePAQ, using the PAQ dataset, achieves accuracy comparable to state-of-the-art retrieve-and-read models like RAG and FiD, while being significantly faster.
  • RePAQ with reranker achieved 47.7% EM on NQ, outperforming recent retrieve-and-read systems by over 3%.
  • RePAQ can be configured for speed (over 1000 Q/sec) or size (under 500MB) with minimal accuracy loss.
  • Highly optimized RePAQ models won tracks in the EfficientQA NeurIPS competition.
  • CBQA Performance: BART models trained on PAQ outperformed baselines by 5% but still lagged RePAQ by over 15%.
  • Selective QA: RePAQ achieved 59% accuracy when answering 75% of questions, outperforming FiD (55%) due to better calibration.
  • Hybrid System: A combined RePAQ + FiD system was 2.1x faster than FiD-large alone, with RePAQ handling 57% of questions and achieving slightly higher overall accuracy.

Limitations and Future Work

  • PAQ Generation Cost: Generating PAQ is computationally intensive due to its large scale.
  • Filtering Model Bound: The ODQA filtering model used for PAQ generation might impose an upper bound on RePAQ's performance, especially if the filtering is less effective on certain datasets (like TriviaQA).
  • CBQA Memorization: CBQA models struggle to effectively memorize the vast knowledge in PAQ, indicating a need for future work on how to better store PAQ in CBQA model parameters.
  • Future Work: The authors suggest exploring PAQ for general data augmentation, improving the efficiency of PAQ generation, and further investigating the trade-offs between parametric and non-parametric knowledge storage. They also propose exploring PAQ to improve retrieve-and-read systems.

Summary updated

Abstract

Open-domain Question Answering models which directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared to conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models lack the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically-generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) whilst retaining high accuracy. Lastly, we demonstrate RePAQ's strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to ``back-off" to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.
BibTeX
@article{lewis2021paq,
  author = {Patrick S. H. Lewis and Yuxiang Wu and Linqing Liu and Pasquale Minervini and Heinrich Küttler and Aleksandra Piktus and Pontus Stenetorp and Sebastian Riedel},
  title = {{PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00415},
}
Figure