Latent Retrieval for Weakly Supervised Open Domain Question Answering

Kenton Lee, Ming-Wei Chang, Kristina Toutanova

Abstract: Kenton Lee, Ming-Wei Chang, Kristina Toutanova. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

Open-domain question answering (QA) requires a system to navigate a massive corpus, such as Wikipedia, to find evidence and extract precise answer strings. Traditionally, this task has been handled by a two-stage pipeline: a fixed Information Retrieval (IR) system (e.g., BM25) that identifies candidate documents based on keyword overlap, followed by a neural "reader" that extracts the answer.

This paper argues that such decoupled pipelines are suboptimal. Keyword-based IR is often insufficient for real-world "information-seeking" questions where the user may not know the specific terminology used in the evidence. Furthermore, because traditional retrievers are non-trainable "black boxes," they cannot be optimized for the specific nuances of the QA task. The authors propose ORQA (Open Retrieval Question Answering), the first system to jointly learn both the retriever and the reader in an end-to-end fashion using only question-answer pairs.

The ORQA Framework

ORQA treats the selection of evidence as a latent variable. Instead of relying on a fixed index, it learns to map both questions and evidence blocks into a shared dense vector space.

The ORQA architecture: A dual-encoder retriever finds evidence blocks, followed by a BERT-based reader that extracts the answer span.

Model Architecture

  • Retriever: Uses two BERT-based encoders. One produces a vector representation for the question ($h_q$), and the other for the evidence block ($h_b$). The retrieval score is the inner product of these two vectors. To handle the scale of 13 million Wikipedia blocks, the system utilizes Maximum Inner Product Search (MIPS) for efficient querying.
  • Reader: A standard BERT-based span extractor. It processes the top-$k$ blocks retrieved by the first stage to identify the start and end indices of the most likely answer string.

Solving the "Cold Start" with Inverse Cloze Task

A major challenge in training a retriever from scratch is the lack of "gold" evidence documents in standard QA datasets; we typically only have the question and the answer string. Without a strong initial retriever, the model is unlikely to find the correct answer by chance, providing no gradient for learning.

To solve this, the authors introduce the Inverse Cloze Task (ICT) for unsupervised pre-training.

The Inverse Cloze Task (ICT) pre-trains the retriever by tasking it to find the context block for a given 'pseudo-question' sentence.

In ICT, a random sentence is sampled from Wikipedia to serve as a "pseudo-question," and the model must retrieve the surrounding context block from a large batch of candidates. This forces the retriever to learn semantic relationships and topical associations beyond simple lexical matching, providing a "sufficiently strong initialization" for subsequent end-to-end fine-tuning on QA pairs.

Experimental Results

ORQA was evaluated on five benchmarks, distinguishing between information-seeking datasets (Natural Questions, WebQuestions, CuratedTrec) and knowledge-testing datasets (SQuAD, TriviaQA).

The results demonstrate that ORQA significantly outperforms BM25-based baselines on information-seeking tasks:

  • WebQuestions: Achieved 36.4 Exact Match (EM), an 18.7 point improvement over the BM25 + BERT baseline.
  • Natural Questions: Achieved 33.3 EM compared to 26.5 for the baseline.

Interestingly, on SQuAD, the BM25 baseline remained superior (33.2 vs 20.2). The authors attribute this to SQuAD's construction: because questions were written by people who had already seen the text, they rely heavily on exact lexical overlaps that keyword-based systems are specifically designed to exploit.

Conclusion and Future Work

ORQA demonstrates that it is possible to learn to retrieve evidence from millions of documents without explicit supervision. By replacing black-box IR with a latent retrieval model, ORQA achieves state-of-the-art performance on real-world queries. Future work may involve hybrid approaches that combine the semantic power of dense retrieval with the precision of sparse keyword matching to handle rare entities and technical terms more effectively.

Summary updated

BibTeX
@inproceedings{lee2019latent,
  author = {Kenton Lee and Ming-Wei Chang and Kristina Toutanova},
  title = {{Latent Retrieval for Weakly Supervised Open Domain Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Lp6-6_vywUu},
}
Figure