Abstract
Open-domain question answering (QA) requires systems to retrieve relevant information from massive corpora before extracting answers. Traditional pipelines often suffer from noisy retrieval where keyword-heavy passages lack the actual answer. We present the Reinforced Ranker-Reader ($R^3$) framework, which jointly trains a neural Ranker and Reader using reinforcement learning. By treating passage selection as a latent action rewarded by answer accuracy, $R^3$ learns to prioritize passages with high semantic relevance over simple keyword overlap. Our model achieves state-of-the-art performance on multiple open-domain QA benchmarks, including Quasar-T and SQuAD-Open.
Introduction
Traditional Reading Comprehension (RC) models typically operate in a "closed-domain" setting, where they are provided with a specific passage containing the answer. In contrast, real-world "open-domain" QA requires searching a massive corpus (e.g., Wikipedia) to find relevant snippets. The standard two-stage pipeline—Information Retrieval (IR) followed by RC—often fails because IR engines like BM25 rely on keyword overlap. A passage might share many keywords with a question but still be irrelevant or contain the wrong information. This "noisy retrieval" problem necessitates a smarter selection mechanism to identify truly "answerable" passages.
The Reinforced Ranker-Reader ($R^3$) Framework
The $R^3$ framework addresses the retrieval gap by introducing two interacting neural components trained via reinforcement learning.
1. The Ranker
The Ranker evaluates the top $N$ passages retrieved by a traditional IR engine. Instead of relying on IR scores, it computes its own probability distribution $\pi(\tau|q)$ over the passages, predicting which one is most likely to contain the correct answer.
2. The Reader
Based on the passage $\tau$ selected by the Ranker, the Reader (an RC model) attempts to extract the specific text span that answers the question. Both components utilize a Match-LSTM architecture, which employs word-level attention to compare the question and passage.
Training via Reinforcement Learning
A key challenge in open-domain QA is the lack of "ground truth" labels for which passage is the best source. $R^3$ overcomes this by using the REINFORCE algorithm. The Ranker's passage selection is treated as a discrete action. If the Reader extracts the correct answer from the selected passage, the Ranker receives a positive reward (based on the F1 score of the extracted span). This feedback loop allows the Ranker to learn semantic nuances that keyword-based IR misses, while the Reader improves its extraction capabilities on the Ranker's prioritized passages.
Experimental Evaluation
The model was evaluated on five diverse datasets: Quasar-T, SQuAD-Open, WikiMovies, CuratedTREC, and WebQuestions. For most datasets, a Lucene-based search index was used to retrieve candidate passages from a Wikipedia dump.
Key Results
$R^3$ demonstrated significant improvements over existing baselines and achieved state-of-the-art results on four of the five benchmarks.
On the Quasar-T dataset, $R^3$ achieved an Exact Match (EM) score of 34.2, significantly outperforming BiDAF (25.9) and GA (26.4). Furthermore, the neural Ranker proved much more effective than raw IR; the top-1 passage selected by $R^3$ contained the correct answer 40.3% of the time, compared to only 19.7% for the initial IR engine.
Discussion and Future Work
The success of $R^3$ highlights that joint training of retrieval and comprehension components is essential for robust open-domain QA. By using F1 scores as a reward signal, the model benefits from a smoother gradient during the reinforcement learning process.
However, the current model is limited by its "hard selection" of a single passage. This makes it unsuitable for multi-hop reasoning tasks where answers must be synthesized from multiple sources. Future research may explore document-level ranking or methods to aggregate information across multiple selected passages to handle more complex queries.
Summary updated