TL;DR
This paper introduces ORQA, the first end-to-end system for open-domain question answering that jointly learns both evidence retrieval and answer reading, using only question-answer pairs for supervision. By pre-training the retriever with an unsupervised Inverse Cloze Task (ICT), ORQA can effectively learn to retrieve relevant information from the entire Wikipedia, outperforming traditional IR methods like BM25 on datasets where users genuinely seek new information.
What Problem Does This Paper Address?
Traditional open-domain question answering (QA) systems typically rely on a two-stage pipeline: first, a separate, "black-box" Information Retrieval (IR) system fetches a set of candidate documents, and second, a reading comprehension model extracts the answer from these candidates. This approach has two main limitations: (1) it assumes the availability of gold evidence for training the reader, which is not always practical, and (2) it treats the IR system as a fixed component, preventing it from being optimized for the downstream QA task. Furthermore, existing weakly supervised methods still often assume noisy gold evidence from IR systems. The authors argue that QA is fundamentally different from IR, requiring deeper language understanding beyond simple lexical matching, and that jointly learning retrieval and reading is crucial for more realistic and effective QA.
Methods
The core innovation is the Open-Retrieval Question Answering (ORQA) model, which learns to retrieve evidence from an open corpus (all of Wikipedia) and read answers from it, all in an end-to-end fashion. The key to making this feasible is a novel pre-training strategy for the retriever using an Inverse Cloze Task (ICT).
- Inverse Cloze Task (ICT): In ICT, a sentence is treated as a "pseudo-question," and its surrounding context is the "pseudo-evidence." The model is trained to select the correct context given the sentence. This unsupervised task mimics evidence retrieval by forcing the model to learn semantic relationships and infer missing information, going beyond simple word matching. To balance learning semantic understanding with lexical overlap, the sentence is masked from its context only 90% of the time.
- Joint Training: ORQA uses BERT as the backbone for both the retriever and reader components. The retriever learns to score evidence blocks based on the inner product of dense vector representations of the question and the block. The reader then scores potential answer spans within the retrieved blocks. The entire model is fine-tuned end-to-end using only question-answer string pairs.
- Inference: To handle the massive search space of Wikipedia, ORQA pre-computes encodings for all evidence blocks. During inference, it uses a beam-search procedure to retrieve the top-k evidence blocks and then applies the reader component.
Datasets
The authors evaluate ORQA on open versions of five existing QA datasets:
- Natural Questions: Questions from Google Search queries.
- WebQuestions: Questions sampled from Google Suggest API.
- CuratedTrec: Question-answer pairs derived from TREC QA data.
- TriviaQA: Trivia question-answer pairs scraped from the web.
- SQuAD: Originally a reading comprehension dataset, adapted for open-domain QA.
They note that for datasets like Natural Questions, WebQuestions, and CuratedTrec, the question askers do not know the answer beforehand, reflecting a more genuine information-seeking scenario. SQuAD and TriviaQA, on the other hand, are written with known answers in mind, which can introduce biases.
Key Insights and Ideas
- QA is not IR: The paper strongly argues that question answering requires deeper language understanding than traditional information retrieval, which often relies on lexical matching. ORQA's success demonstrates the benefit of learning retrieval specifically for QA.
- Unsupervised Pre-training is Key: The Inverse Cloze Task (ICT) is a crucial enabler, allowing the retriever to learn useful representations without requiring labeled evidence. This makes end-to-end learning feasible.
- Learned Retrieval Matters for Information Seeking: ORQA significantly outperforms traditional IR baselines on datasets where users are genuinely seeking information (i.e., question writers don't know the answer). This highlights the importance of a learned, adaptable retriever for real-world QA.
- Dataset Biases Impact Retrieval: The paper points out that datasets where answers are known beforehand (like SQuAD) can lead to artificially high performance for retrieval methods due to inherent biases, making them less suitable for evaluating truly learned retrieval.
Key Results
- ORQA achieves significant improvements over BM25 on datasets where users are seeking information, with gains of 6 to 19 points in exact match.
- On datasets where the questioner already knows the answer (SQuAD, TriviaQA), traditional IR systems like BM25 are competitive, suggesting that for such tasks, precise lexical matching is sufficient.
- The ICT pre-training is shown to be critical; without it, performance drops significantly, especially when too much masking is applied, preventing the model from learning n-gram overlap. Conversely, no masking leads to memorization.
Limitations and Future Work
- 128-Dimensional Vectors: The paper acknowledges that the limited dimensionality of the learned representations (128 dimensions) can be a bottleneck for representing extremely specific concepts, where sparse representations might perform better.
- Hybrid Approaches: The authors suggest that a hybrid approach, combining the strengths of learned retrieval with traditional IR or other methods, could be a promising direction for future work.
- Dataset Specificity: The performance differences across datasets highlight the impact of dataset biases and the need for careful evaluation.
Summary updated