R 3 : Reinforced Ranker-Reader for Open-Domain Question Answering.

Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, Jing Jiang

Abstract

Open-domain question answering (QA) requires systems to retrieve relevant information from massive corpora before extracting answers. Traditional pipelines often suffer from noisy retrieval where keyword-heavy passages lack the actual answer. We present the Reinforced Ranker-Reader ($R^3$) framework, which jointly trains a neural Ranker and Reader using reinforcement learning. By treating passage selection as a latent action rewarded by answer accuracy, $R^3$ learns to prioritize passages with high semantic relevance over simple keyword overlap. Our model achieves state-of-the-art performance on multiple open-domain QA benchmarks, including Quasar-T and SQuAD-Open.

Introduction

Traditional Reading Comprehension (RC) models typically operate in a "closed-domain" setting, where they are provided with a specific passage containing the answer. In contrast, real-world "open-domain" QA requires searching a massive corpus (e.g., Wikipedia) to find relevant snippets. The standard two-stage pipeline—Information Retrieval (IR) followed by RC—often fails because IR engines like BM25 rely on keyword overlap. A passage might share many keywords with a question but still be irrelevant or contain the wrong information. This "noisy retrieval" problem necessitates a smarter selection mechanism to identify truly "answerable" passages.

The Reinforced Ranker-Reader ($R^3$) Framework

The $R^3$ framework addresses the retrieval gap by introducing two interacting neural components trained via reinforcement learning.

The R3 architecture overview, showing the joint training of the Ranker and Reader components through a reward signal.

1. The Ranker

The Ranker evaluates the top $N$ passages retrieved by a traditional IR engine. Instead of relying on IR scores, it computes its own probability distribution $\pi(\tau|q)$ over the passages, predicting which one is most likely to contain the correct answer.

2. The Reader

Based on the passage $\tau$ selected by the Ranker, the Reader (an RC model) attempts to extract the specific text span that answers the question. Both components utilize a Match-LSTM architecture, which employs word-level attention to compare the question and passage.

Training via Reinforcement Learning

A key challenge in open-domain QA is the lack of "ground truth" labels for which passage is the best source. $R^3$ overcomes this by using the REINFORCE algorithm. The Ranker's passage selection is treated as a discrete action. If the Reader extracts the correct answer from the selected passage, the Ranker receives a positive reward (based on the F1 score of the extracted span). This feedback loop allows the Ranker to learn semantic nuances that keyword-based IR misses, while the Reader improves its extraction capabilities on the Ranker's prioritized passages.

Experimental Evaluation

The model was evaluated on five diverse datasets: Quasar-T, SQuAD-Open, WikiMovies, CuratedTREC, and WebQuestions. For most datasets, a Lucene-based search index was used to retrieve candidate passages from a Wikipedia dump.

Key Results

$R^3$ demonstrated significant improvements over existing baselines and achieved state-of-the-art results on four of the five benchmarks.

Comparison of R3 against baseline models across five open-domain QA datasets. R3 consistently outperforms traditional IR-RC pipelines.

On the Quasar-T dataset, $R^3$ achieved an Exact Match (EM) score of 34.2, significantly outperforming BiDAF (25.9) and GA (26.4). Furthermore, the neural Ranker proved much more effective than raw IR; the top-1 passage selected by $R^3$ contained the correct answer 40.3% of the time, compared to only 19.7% for the initial IR engine.

Discussion and Future Work

The success of $R^3$ highlights that joint training of retrieval and comprehension components is essential for robust open-domain QA. By using F1 scores as a reward signal, the model benefits from a smoother gradient during the reinforcement learning process.

However, the current model is limited by its "hard selection" of a single passage. This makes it unsuitable for multi-hop reasoning tasks where answers must be synthesized from multiple sources. Future research may explore document-level ranking or methods to aggregate information across multiple selected passages to handle more complex queries.

Summary updated

BibTeX
@inproceedings{wang2018r,
  author = {Shuohang Wang and Mo Yu and Xiaoxiao Guo and Zhiguo Wang and Tim Klinger and Wei Zhang and Shiyu Chang and Gerald Tesauro and Bowen Zhou and Jing Jiang},
  title = {{R 3 : Reinforced Ranker-Reader for Open-Domain Question Answering.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12053},
}
Figure