Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs

Ye Liu, Rui Meng, Meghana Moorthy Bhat, Shafiq Joty, Caiming Xiong, Yingbo Zhou, Semih Yavuz

Abstract

Standard Retrieval-Augmented Generation (RAG) often fails because Large Language Models (LLMs) become overwhelmed by long contexts, frequently returning "unknown" even when the answer is present in the retrieved documents. This work introduces a multi-round strategy that treats "unknown" responses as signals of model uncertainty. By using a "Post-Fusion" fallback mechanism, the proposed approach significantly boosts accuracy and reduces errors across major open-domain QA benchmarks.

Introduction

Retrieval-Augmented Generation (RAG) is the industry standard for grounding Large Language Models (LLMs) in external facts. Typically, a Concatenation strategy is employed: the top-$k$ retrieved documents are prepended to the prompt to provide context for the LLM's answer.

However, researchers have identified a significant flaw: LLMs often suffer from "distraction" or "lost-in-the-middle" effects. When multiple passages are concatenated, the model frequently returns an "unknown" response, even when the correct answer is guaranteed to be within the retrieved set. An alternative, Post-Fusion (querying the LLM for each passage individually and using a majority vote), reduces "unknowns" but introduces a different risk: the majority might converge on a common incorrect answer.

Figure 1: Comparison of Concatenation and Post-Fusion strategies. Concatenation often leads to "unknown" responses as the number of passages increases, while Post-Fusion struggles with non-majority correct answers.

Proposed Methodology

To bridge the gap between these strategies, the authors explored several multi-round approaches that leverage model uncertainty.

Single-Round Baselines

  • Zero-shot Concatenation & Post-Fusion: Standard baseline methods.
  • Few-shot Pruning: Uses Chain-of-Thought (CoT) prompting to teach the model to discard irrelevant passages before answering.
  • Few-shot Summarization: Teaches the model to summarize relevant information from passages before deriving an answer.

Multi-Round Strategies

The core contribution of this work lies in two multi-round strategies:

  1. Post-Fusion as the Fallback (Concat + PF): The system first attempts the efficient Concatenation method. If the model returns "unknown," it triggers a second round using Post-Fusion (individual passage processing followed by a majority vote).
  2. Concatenation as the Distiller (PF + Concat): The system first generates a pool of candidate answers via Post-Fusion. In the second round, it concatenates only the "successful" passages (those that did not yield "unknown") and the candidates to produce a final answer.
Figure 2: Architectures for the proposed multi-round strategies: Post-Fusion as the Fallback (top) and Concatenation as the Distiller (bottom).

Experimental Evaluation

The methods were evaluated on three open-domain question-answering benchmarks: Natural Questions (NQ), TriviaQA, and SQuAD-Open. Testing was conducted across three LLMs: Llama2-7b-chat, GPT-3.5, and GPT-4.

Key Insights

  • "Unknown" as a Signal: The paper treats an LLM's "unknown" response as a form of self-aware uncertainty, using it as a trigger for more robust, albeit computationally expensive, reasoning.
  • Robustness to Context Length: While standard concatenation performance drops as the number of passages ($k$) increases due to noise, multi-round approaches remain stable or improve.
  • Efficiency: The "Fallback" (Concat + PF) method is highly efficient, invoking the multi-call Post-Fusion approach only when necessary.

Results

The multi-round approaches demonstrated significant improvements:

  • Performance Gains: Outperformed traditional concatenation by over 10% in Exact Match (EM) scores across all datasets.
  • Reduced Uncertainty: In the NQ dataset with GPT-3.5, the "unknown" rate was slashed from ~20% to under 7% using the Concat + PF strategy.
  • SOTA Parity: Using GPT-4 with the multi-round strategy achieved performance on par with supervised models.

Conclusion

By treating "unknown" responses as actionable signals of uncertainty rather than simple failures, this research provides a robust framework for improving RAG. The proposed fallback mechanism balances efficiency and accuracy, mitigating the "lost-in-the-middle" effect common in long-context LLM applications.

Summary updated

Abstract

The integration of retrieved passages and large language models (LLMs), such as ChatGPTs, has significantly contributed to improving open-domain question answering. However, there is still a lack of exploration regarding the optimal approach for incorporating retrieved passages into the answer generation process. This paper aims to fill this gap by investigating different methods of combining retrieved passages with LLMs to enhance answer generation. We begin by examining the limitations of a commonly-used concatenation approach. Surprisingly, this approach often results in generating "unknown" outputs, even when the correct document is among the top-k retrieved passages. To address this issue, we explore four alternative strategies for integrating the retrieved passages with the LLMs. These strategies include two single-round methods that utilize chain-of-thought reasoning and two multi-round strategies that incorporate feedback loops. Through comprehensive analyses and experiments, we provide insightful observations on how to effectively leverage retrieved passages to enhance the answer generation capability of LLMs.
BibTeX
@misc{liu2023modeling,
  author = {Ye Liu and Rui Meng and Meghana Moorthy Bhat and Shafiq Joty and Caiming Xiong and Yingbo Zhou and Semih Yavuz},
  title = {{Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs}},
  year = {2023},
  eprint = {2308.12574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12574},
}
Figure