TL;DR
Standard retrieval-augmented generation (RAG) evaluations often conflate retrieval failures with reader formatting issues, making it hard to diagnose why a system underperforms. By introducing the concept of "evidence interfaces" and tracking whether the entire multi-hop reasoning chain survives truncation, this paper reveals that aggregate top-$k$ metrics mask drastically different reader behaviors.
What Problem Does This Paper Address?
In Retrieval-Augmented Generation (RAG), a retriever fetches relevant documents, and a reader (typically an LLM) uses them to generate an answer. When a RAG system fails, developers usually look at aggregate metrics like top-$k$ F1 scores. However, a low score is highly ambiguous:
- Availability Failure: Did the retriever fail to find a crucial piece of the evidence chain?
- Usability Failure: Is the evidence present, but formatted or ordered in a way that the reader model struggles to process?
This paper tackles this measurement problem. Without separating these two failure modes, system designers risk optimizing the wrong component—such as swapping out a retriever when the reader simply needs a better prompt layout, or vice versa.
Methods
The authors formalize the reader-facing presentation of text as an evidence interface—a renderer that selects, orders, and structures evidence before it hits the reader.
To cleanly separate availability from usability, they introduce a diagnostic protocol:
- Matched Adaptation: Instead of just swapping prompts at inference time, they train separate reader models (using parameter-efficient LoRA adapters) on specific evidence interfaces. This ensures the reader is fully adapted to the format it is evaluated on.
- Complete-Support Recall (CSR@k): A strict metric that evaluates whether every single unit of a multi-hop evidence chain is present in the top-$k$ window.
- Coverage-Conditioned Analysis: They split the evaluation dataset into two bins: one where the full support chain survives truncation, and one where it is broken.
The framework compares several interface families, ranging from raw un-ordered context to "support-first" layouts (moving gold evidence to the front) and oracle text.
Datasets
The study evaluates RAG performance on three popular multi-hop question-answering benchmarks that feature explicit sentence- or paragraph-level support annotations:
- HotpotQA
- 2WikiMultiHopQA
- MuSiQue
These datasets are ideal because answering a question correctly requires connecting multiple distinct pieces of information scattered across distractor texts.
Key Insights and Ideas
- The Top-$k$ Blind Spot: Aggregate top-$k$ scores average out two completely opposite phenomena. When the full support chain survives in a short window, the compression actually helps the reader. When even one link is missing, performance plummets.
- Multi-Hop is All-or-Nothing: Standard retrieval evaluations often use "mean support fraction" (giving partial credit for finding some documents). The authors show this is deceptive for multi-hop QA: missing the final hop makes the question completely unanswerable, meaning partial retrieval is often as bad as zero retrieval.
- Layout Dictates Learning: How evidence is ordered during training fundamentally changes what the reader learns. Moving the necessary support to the beginning of the prompt ("support-first") consistently yields better-adapted readers, proving that "lost-in-the-middle" effects are a critical training-time factor, not just an inference quirk.
Key Results
The authors split 1,000 validation examples by whether a ranker-specific top-5 (@5) window captured the full support chain:
- The Split Effect: On the MuSiQue dataset, evaluating the
@5window as an aggregate score shows a massive loss of -0.132 F1 compared to raw context. However, when conditioned on coverage, the@5window actually improves F1 by +0.065 when full support is present, and drops by -0.222 when support is missing. - Support-First Headroom: Simply moving the gold support paragraphs to the front of the text (
SF) while keeping distractors improved F1 across all datasets (+0.026 on HotpotQA, +0.039 on 2Wiki, and +0.070 on MuSiQue), demonstrating significant reader-side headroom.
Limitations and Future Work
- Annotation Dependency: The diagnostic framework relies entirely on datasets with gold support annotations. In many real-world production RAG pipelines, such fine-grained labels do not exist.
- Closed Candidate Pools: The study focuses on re-ranking and formatting within a fixed, closed pool of documents rather than open-web retrieval.
- Future Direction: The authors suggest moving away from gold-annotated diagnostics toward training learned interface builders—components explicitly optimized to arrange and compress text in ways that maximize usability for a specific target reader model.
Summary updated