Abstract: Kenton Lee, Ming-Wei Chang, Kristina Toutanova. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
2019.
Introduction
Open-domain question answering (QA) requires a system to navigate a massive corpus, such as Wikipedia, to find evidence and extract precise answer strings. Traditionally, this task has been handled by a two-stage pipeline: a fixed Information Retrieval (IR) system (e.g., BM25) that identifies candidate documents based on keyword overlap, followed by a neural "reader" that extracts the answer.
This paper argues that such decoupled pipelines are suboptimal. Keyword-based IR is often insufficient for real-world "information-seeking" questions where the user may not know the specific terminology used in the evidence. Furthermore, because traditional retrievers are non-trainable "black boxes," they cannot be optimized for the specific nuances of the QA task. The authors propose ORQA (Open Retrieval Question Answering), the first system to jointly learn both the retriever and the reader in an end-to-end fashion using only question-answer pairs.
The ORQA Framework
ORQA treats the selection of evidence as a latent variable. Instead of relying on a fixed index, it learns to map both questions and evidence blocks into a shared dense vector space.
Model Architecture
- Retriever: Uses two BERT-based encoders. One produces a vector representation for the question ($h_q$), and the other for the evidence block ($h_b$). The retrieval score is the inner product of these two vectors. To handle the scale of 13 million Wikipedia blocks, the system utilizes Maximum Inner Product Search (MIPS) for efficient querying.
- Reader: A standard BERT-based span extractor. It processes the top-$k$ blocks retrieved by the first stage to identify the start and end indices of the most likely answer string.
Solving the "Cold Start" with Inverse Cloze Task
A major challenge in training a retriever from scratch is the lack of "gold" evidence documents in standard QA datasets; we typically only have the question and the answer string. Without a strong initial retriever, the model is unlikely to find the correct answer by chance, providing no gradient for learning.
To solve this, the authors introduce the Inverse Cloze Task (ICT) for unsupervised pre-training.
In ICT, a random sentence is sampled from Wikipedia to serve as a "pseudo-question," and the model must retrieve the surrounding context block from a large batch of candidates. This forces the retriever to learn semantic relationships and topical associations beyond simple lexical matching, providing a "sufficiently strong initialization" for subsequent end-to-end fine-tuning on QA pairs.
Experimental Results
ORQA was evaluated on five benchmarks, distinguishing between information-seeking datasets (Natural Questions, WebQuestions, CuratedTrec) and knowledge-testing datasets (SQuAD, TriviaQA).
The results demonstrate that ORQA significantly outperforms BM25-based baselines on information-seeking tasks:
- WebQuestions: Achieved 36.4 Exact Match (EM), an 18.7 point improvement over the BM25 + BERT baseline.
- Natural Questions: Achieved 33.3 EM compared to 26.5 for the baseline.
Interestingly, on SQuAD, the BM25 baseline remained superior (33.2 vs 20.2). The authors attribute this to SQuAD's construction: because questions were written by people who had already seen the text, they rely heavily on exact lexical overlaps that keyword-based systems are specifically designed to exploit.
Conclusion and Future Work
ORQA demonstrates that it is possible to learn to retrieve evidence from millions of documents without explicit supervision. By replacing black-box IR with a latent retrieval model, ORQA achieves state-of-the-art performance on real-world queries. Future work may involve hybrid approaches that combine the semantic power of dense retrieval with the precision of sparse keyword matching to handle rare entities and technical terms more effectively.
Summary updated