A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models

Gongbo Zhang, Zihan Xu, Qiao Jin, Fangyi Chen, Yilu Fang, Yi Liu, Justin F. Rousseau, Ziyang Xu, Zhiyong Lu, Chunhua Weng, Yifan PengarXiv 2024 · OpenReview

Abstract

While holding great promise for improving and facilitating healthcare, large language models (LLMs) struggle to produce up-to-date responses on evolving topics due to outdated knowledge or hallucination. Retrieval-augmented generation (RAG) is a pivotal innovation that improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge. However, the quality of RAG responses can be largely impacted by the rank and density of key information in the retrieval results, such as the "lost-in-the-middle" problem. In this work, we aim to improve the robustness and reliability of the RAG workflow in the medical domain. Specifically, we propose a map-reduce strategy, BriefContext, to combat the "lost-in-the-middle" issue without modifying the model weights. We demonstrated the advantage of the workflow with various LLM backbones and on multiple QA datasets. This method promises to improve the safety and reliability of LLMs deployed in healthcare domains.

Introduction: The "Lost-in-the-Middle" Challenge

Retrieval-Augmented Generation (RAG) has become the standard for providing Large Language Models (LLMs) with up-to-date knowledge. However, LLMs often exhibit a "U-shaped" attention curve: they excel at utilizing information at the beginning or end of a prompt but frequently ignore key details buried in the middle. This phenomenon, known as the "lost-in-the-middle" problem, poses significant safety risks in high-stakes fields like healthcare, where missing a critical piece of evidence can lead to hallucinations or incomplete clinical advice.

Methods: The BriefContext Pipeline

To address this, researchers introduced BriefContext, a model-agnostic framework that treats long-context reasoning as a distributed task. The pipeline consists of four distinct stages:

  1. Retrieval: Documents are fetched from external knowledge bases (e.g., PubMed abstracts and medical textbooks) using search engines like MedCPT.
  2. Preflight Check: To optimize computational costs, the system compares rankings from different algorithms (MedCPT and BM25). If the top results diverge significantly (low Intersection-over-Union), the system triggers the MapReduce workflow to prevent potential information loss.
  3. Context Map: The retrieved documents are partitioned into smaller chunks. Each partition is processed in parallel by an LLM to extract only the information relevant to the user's query.
  4. Context Reduce: The extracted snippets are aggregated and sent to a final LLM session to synthesize a coherent, final answer.
The BriefContext workflow pipeline, illustrating the four-stage process from user query to final answer.

Key Results and Performance

BriefContext was evaluated using the MIRAGE benchmark, which includes medical examination subsets (MedQA-US, MedMCQA) and biomedical research datasets (PubMedQA). The framework consistently outperformed standard RAG across multiple LLM backbones:

  • Llama2-70B-chat: Accuracy improved from 55.81% to 66.47%.
  • Mixtral-7x8b: Accuracy rose from 70.52% to 72.20%.
  • GPT-3.5-Turbo: Demonstrated significant robustness, particularly when critical information was located at the 50th percentile of the context window.
Comparison of Accuracy between standard RAG and BriefContext (BC) across different key document positions (0th to 100th percentile).

As shown in the experimental data, standard RAG performance often dips significantly when relevant information is placed in the middle of the context. BriefContext effectively flattens this "U-shaped" curve, maintaining high accuracy regardless of the information's position within the retrieved results.

Discussion and Insights

The study highlights that LLMs are fundamentally more effective at reasoning over several short contexts than one massive one. Furthermore, the "Preflight Check" proved highly efficient, achieving a 92.61% recall in identifying potential RAG failures. While BriefContext introduces a slight increase in token usage (approximately 10%) and latency due to the parallel "Map" steps, the gains in reliability and accuracy provide a necessary safety margin for clinical applications.

Conclusion

BriefContext offers a robust solution to the positional bias inherent in current LLMs. By leveraging a MapReduce-inspired architecture, it improves the reliability of RAG systems in the medical domain without the need for expensive model fine-tuning. Future work will explore its applicability in other specialized domains such as legal and technical documentation.

Summary updated

Abstract

While holding great promise for improving and facilitating healthcare, large language models (LLMs) struggle to produce up-to-date responses on evolving topics due to outdated knowledge or hallucination. Retrieval-augmented generation (RAG) is a pivotal innovation that improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge. However, the quality of RAG responses can be largely impacted by the rank and density of key information in the retrieval results, such as the "lost-in-the-middle" problem. In this work, we aim to improve the robustness and reliability of the RAG workflow in the medical domain. Specifically, we propose a map-reduce strategy, BriefContext, to combat the "lost-in-the-middle" issue without modifying the model weights. We demonstrated the advantage of the workflow with various LLM backbones and on multiple QA datasets. This method promises to improve the safety and reliability of LLMs deployed in healthcare domains.
BibTeX
@misc{zhang2024mapreduce,
  author = {Gongbo Zhang and Zihan Xu and Qiao Jin and Fangyi Chen and Yilu Fang and Yi Liu and Justin F. Rousseau and Ziyang Xu and Zhiyong Lu and Chunhua Weng and Yifan Peng},
  title = {{A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models}},
  year = {2024},
  eprint = {2412.15271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15271},
}
Figure