Abstract
Open-Domain Question Answering (ODQA) systems often struggle with the quality of retrieved passages, which may contain conflicting information and be misaligned with the reader’s needs. Existing retrieval methods aim to gather relevant passages but often fail to prioritize consistent and useful information for the reader. In this paper, we introduce a novel Reader-Centered Passage Selection (R-CPS) method, which enhances the performance of the retrieve-then-read pipeline by re-ranking and clustering passages from the reader’s perspective. Our method re-ranks passages based on the reader’s prediction probability distribution and clusters passages according to the predicted answers, prioritizing more useful and relevant passages to the top and reducing inconsistent information. Experiments on ODQA datasets demonstrate the effectiveness of our approach in improving the quality of evidence passages under zero-shot settings.
Introduction
In Open-Domain Question Answering (ODQA), the standard "retrieve-then-read" pipeline has become the foundational architecture. This process involves retrieving a set of relevant documents from a massive corpus (such as Wikipedia) and providing them to a Large Language Model (LLM), or "reader," to synthesize an answer. However, a significant bottleneck remains: retrievers often prioritize passages based on semantic similarity rather than actual utility, leading to the inclusion of conflicting or irrelevant information that confuses the reader.
The Challenge of Inconsistent Evidence
Most ODQA systems suffer from two primary issues:
- Inconsistent Evidence: Retrievers may return multiple passages that offer different answers to the same question. For example, when asked when Toyota first came to the U.S., different passages might cite 1957, 1958, or 1986, creating noise that makes it difficult for the reader to identify the ground truth.
- Preference Divergence: Retrievers typically rank passages using vector distance (semantic similarity). However, a passage can be semantically similar to a query without being useful for answering it. This misalignment between the retriever's "preferences" and the reader's "needs" degrades final performance.
The R-CPS Framework
To address these challenges, we propose Reader-Centered Passage Selection (R-CPS). This method allows the reader model to evaluate the utility of passages before the final synthesis stage. The framework consists of three distinct stages:
1. Answer Prediction
Instead of processing all passages simultaneously, the reader first attempts to extract a candidate answer from each passage individually. If a passage lacks sufficient information, the model is instructed to output "unknown."
2. Reader-Centered Passage Re-ranking (RCPR)
The system calculates a relevance score based on the probability that the reader did not return "unknown." This effectively measures the "usability" of a passage from the reader's specific perspective.
3. Reader-Centered Passage Clustering (RCPC)
Passages are grouped based on the candidate answers they yielded. The system then scores these clusters by summing the relevance scores of their constituent passages. By feeding the reader passages from the highest-scoring clusters, the system ensures the evidence is consistent and supported by multiple sources.
Experimental Evaluation
We evaluated R-CPS on three standard ODQA benchmarks: Natural Questions (NQ), WebQuestions (WebQ), and TriviaQA. The method was tested against various retrieval backends, including BM25 (sparse), DPR, and Contriever (dense).
Key Results
- Accuracy Gains: On the Natural Questions dataset using BM25, R-CPS improved the Exact Match (EM) score by 10 points compared to the baseline pipeline.
- Superior Re-ranking: R-CPS consistently outperformed popular re-ranking methods like UPR (Unsupervised Passage Re-ranker) and BGE. For instance, on TriviaQA with Contriever, R-CPS achieved an EM of 56.1, surpassing UPR’s 54.9.
- Consensus Filtering: By clustering by answer, the method naturally suppresses "distractor" passages. Even if a distractor is highly ranked by a retriever, it is unlikely to have a high cluster score unless multiple other passages support its incorrect claim.
Conclusion
R-CPS demonstrates that the LLM itself is often the best judge of its own input. By shifting the focus from how well a passage matches a question to how well it serves the reader's needs, we can significantly reduce noise and improve the reliability of ODQA systems. Future work will explore more advanced clustering techniques and the application of reader-centered selection to long-form generation tasks.
Summary updated