Abstract
Standard Retrieval-Augmented Generation (RAG) often fails because Large Language Models (LLMs) become overwhelmed by long contexts, frequently returning "unknown" even when the answer is present in the retrieved documents. This work introduces a multi-round strategy that treats "unknown" responses as signals of model uncertainty. By using a "Post-Fusion" fallback mechanism, the proposed approach significantly boosts accuracy and reduces errors across major open-domain QA benchmarks.
Introduction
Retrieval-Augmented Generation (RAG) is the industry standard for grounding Large Language Models (LLMs) in external facts. Typically, a Concatenation strategy is employed: the top-$k$ retrieved documents are prepended to the prompt to provide context for the LLM's answer.
However, researchers have identified a significant flaw: LLMs often suffer from "distraction" or "lost-in-the-middle" effects. When multiple passages are concatenated, the model frequently returns an "unknown" response, even when the correct answer is guaranteed to be within the retrieved set. An alternative, Post-Fusion (querying the LLM for each passage individually and using a majority vote), reduces "unknowns" but introduces a different risk: the majority might converge on a common incorrect answer.
Proposed Methodology
To bridge the gap between these strategies, the authors explored several multi-round approaches that leverage model uncertainty.
Single-Round Baselines
- Zero-shot Concatenation & Post-Fusion: Standard baseline methods.
- Few-shot Pruning: Uses Chain-of-Thought (CoT) prompting to teach the model to discard irrelevant passages before answering.
- Few-shot Summarization: Teaches the model to summarize relevant information from passages before deriving an answer.
Multi-Round Strategies
The core contribution of this work lies in two multi-round strategies:
- Post-Fusion as the Fallback (Concat + PF): The system first attempts the efficient Concatenation method. If the model returns "unknown," it triggers a second round using Post-Fusion (individual passage processing followed by a majority vote).
- Concatenation as the Distiller (PF + Concat): The system first generates a pool of candidate answers via Post-Fusion. In the second round, it concatenates only the "successful" passages (those that did not yield "unknown") and the candidates to produce a final answer.
Experimental Evaluation
The methods were evaluated on three open-domain question-answering benchmarks: Natural Questions (NQ), TriviaQA, and SQuAD-Open. Testing was conducted across three LLMs: Llama2-7b-chat, GPT-3.5, and GPT-4.
Key Insights
- "Unknown" as a Signal: The paper treats an LLM's "unknown" response as a form of self-aware uncertainty, using it as a trigger for more robust, albeit computationally expensive, reasoning.
- Robustness to Context Length: While standard concatenation performance drops as the number of passages ($k$) increases due to noise, multi-round approaches remain stable or improve.
- Efficiency: The "Fallback" (Concat + PF) method is highly efficient, invoking the multi-call Post-Fusion approach only when necessary.
Results
The multi-round approaches demonstrated significant improvements:
- Performance Gains: Outperformed traditional concatenation by over 10% in Exact Match (EM) scores across all datasets.
- Reduced Uncertainty: In the NQ dataset with GPT-3.5, the "unknown" rate was slashed from ~20% to under 7% using the Concat + PF strategy.
- SOTA Parity: Using GPT-4 with the multi-round strategy achieved performance on par with supervised models.
Conclusion
By treating "unknown" responses as actionable signals of uncertainty rather than simple failures, this research provides a robust framework for improving RAG. The proposed fallback mechanism balances efficiency and accuracy, mitigating the "lost-in-the-middle" effect common in long-context LLM applications.
Summary updated