Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs

Liu Ye, Semih Yavuz, Rui Meng, M Vinayaga Moorthy, Shafiq Joty, Caiming Xiong, Yingbo Zhou

Abstract

The integration of retrieved passages and large language models (LLMs), such as ChatGPTs, has significantly contributed to improving open-domain question answering. However, there is still a lack of exploration regarding the optimal approach for incorporating retrieved passages into the answer generation process. This paper aims to fill this gap by investigating different methods of combining retrieved passages with LLMs to enhance answer generation. We begin by examining the limitations of a commonly-used concatenation approach. Surprisingly, this approach often results in generating "unknown" outputs, even when the correct document is among the top-k retrieved passages. To address this issue, we explore four alternative strategies for integrating the retrieved passages with the LLMs. These strategies include two single-round methods that utilize chain-of-thought reasoning and two multi-round strategies that incorporate feedback loops. Through comprehensive analyses and experiments, we provide insightful observations on how to effectively leverage retrieved passages to enhance the answer generation capability of LLMs.

Introduction

Retrieval Augmented Generation (RAG) is the current standard for grounding Large Language Models (LLMs) in external facts. The typical workflow involves retrieving the top-$k$ passages from a search engine, concatenating them into a single prompt, and asking the LLM for an answer. However, this paper identifies a glaring weakness: LLMs often get overwhelmed by long contexts. Even when the correct answer is present in the retrieved documents, models frequently suffer from "contextual confusion" and output "unknown."

The "Unknown" Paradox in Concatenation

The authors found that standard concatenation leads to "unknown" responses roughly 20% of the time. This occurs even in an "oracle" setting where the correct document is known to be in the top-k retrieved passages. This suggests that LLMs are much better at finding information in small, individual snippets than in one large, concatenated block.

Comparison of Concatenation and Post-Fusion architectures and their respective 'Unknown' response rates.

As illustrated in the figure above, while Concatenation is efficient, its error rate is high. Conversely, Post-Fusion—where each document is processed individually and a majority vote determines the final answer—drops the "unknown" rate to between 2% and 7%. However, Post-Fusion can fail if the correct answer is present but does not win the majority vote due to the presence of many irrelevant documents.

Proposed Multi-Round Strategies

To bridge the gap between efficiency and robustness, the researchers investigated several strategies, focusing on multi-round approaches that treat the model's output as a signal for further processing.

Post-Fusion as Fallback (Concat+PF)

This is the core contribution of the paper. The system first attempts the efficient Concatenation method. If the model returns "unknown," it is treated as a measure of uncertainty, triggering a second round using Post-Fusion. This fallback mechanism balances speed with the robustness of individual passage processing.

Concatenation as the Distiller (PF+Concat)

This strategy uses Post-Fusion in the first stage to generate a pool of candidate answers and identify relevant passages. It then concatenates only these filtered, high-quality passages for a final generation round.

Architectural diagrams for the proposed multi-round strategies: Post-Fusion as Fallback and Concatenation as Distiller.

Key Results and Insights

The authors evaluated these methods on three classic benchmarks: Natural Questions (NQ), TriviaQA, and SQuAD-Open.

  • Accuracy Boost: On the NQ dataset, the Concat + PF approach outperformed standard concatenation by over 10% in Exact Match (EM) scores across Llama2, ChatGPT, and GPT-4.
  • Structural Superiority: Multi-round architectural changes consistently outperformed single-round "prompt engineering" methods like Chain-of-Thought pruning or summarization.
  • Token Efficiency: While multi-round methods sound expensive, the "Fallback" approach only triggers the second round when the first fails, making it more cost-effective than pure Post-Fusion while maintaining high accuracy.
  • Position Matters: The study corroborated the "U-shaped" performance curve, finding that placing the relevant document at the end of the list (closest to the question) improved performance.

Conclusion

By modeling uncertainty through the "unknown" response, RAG systems can intelligently switch between efficient concatenation and robust post-fusion. This hybrid approach significantly improves the reliability of LLMs in open-domain question answering without incurring the full cost of processing every document individually for every query.

Summary updated

Abstract

The integration of retrieved passages and large language models (LLMs), such as ChatGPTs, has significantly contributed to improving open-domain question answering. However, there is still a lack of exploration regarding the optimal approach for incorporating retrieved passages into the answer generation process. This paper aims to fill this gap by investigating different methods of combining retrieved passages with LLMs to enhance answer generation. We begin by examining the limitations of a commonly-used concatenation approach. Surprisingly, this approach often results in generating "unknown" outputs, even when the correct document is among the top-k retrieved passages. To address this issue, we explore four alternative strategies for integrating the retrieved passages with the LLMs. These strategies include two single-round methods that utilize chain-of-thought reasoning and two multi-round strategies that incorporate feedback loops. Through comprehensive analyses and experiments, we provide insightful observations on how to effectively leverage retrieved passages to enhance the answer generation capability of LLMs.
BibTeX
@misc{ye2023modeling,
  author = {Liu Ye and Semih Yavuz and Rui Meng and M Vinayaga Moorthy and Shafiq Joty and Caiming Xiong and Yingbo Zhou},
  title = {{Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs}},
  year = {2023},
  eprint = {2308.12574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12574},
}
Figure