How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs' internal prior

Kevin Wu, Eric Wu, James Zou

Abstract

Retrieval augmented generation (RAG) is frequently used to mitigate hallucinations and provide up-to-date knowledge for large language models (LLMs). However, given that document retrieval is an imprecise task and sometimes results in erroneous or even harmful content being presented in context, this raises the question of how LLMs handle retrieved information: If the provided content is incorrect, does the model know to ignore it, or does it recapitulate the error? Conversely, when the model's initial response is incorrect, does it always know to use the retrieved information to correct itself, or does it insist on its wrong prior response? To answer this, we curate a dataset of over 1200 questions across six domains (e.g., drug dosages, Olympic records, locations) along with content relevant to answering each question. We further apply precise perturbations to the answers in the content that range from subtle to blatant errors. We benchmark six top-performing LLMs, including GPT-4o, on this dataset and find that LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time. However, the more unrealistic the retrieved content is (i.e. more deviated from truth), the less likely the model is to adopt it. Also, the less confident a model is in its initial response (via measuring token probabilities), the more likely it is to adopt the information in the retrieved content. We exploit this finding and demonstrate simple methods for improving model accuracy where there is conflicting retrieved content. Our results highlight a difficult task and benchmark for LLMs -- namely, their ability to correctly discern when it is wrong in light of correct retrieved content and to reject cases when the provided content is incorrect.

Introduction: The RAG Arbitration Problem

Retrieval-Augmented Generation (RAG) has become the industry standard for grounding Large Language Models (LLMs) in factual, up-to-date information. By providing external documents in the model's context window, developers aim to reduce hallucinations. However, this approach assumes that the retrieved documents are inherently accurate. In real-world applications, retrieval systems often return outdated information, satirical content, or factual errors. This creates a "tug-of-war" between the model's internal knowledge (its prior) and the external evidence provided in the context. A robust model must act as an effective arbitrator: it should trust the context when its internal knowledge is lacking or incorrect, but it must also be capable of rejecting context when it is clearly erroneous.

The ClashEval Benchmark

To quantify this arbitration process, we introduced ClashEval, a dataset comprising over 1,200 questions across six diverse domains. These domains were chosen to test different types of knowledge, including numerical, temporal, and categorical data:

  • Drug Dosage: Critical numerical values sourced from UpToDate.com.
  • News: Recent events from Associated Press feeds.
  • Wikipedia Dates: Historical years and timelines.
  • Sports Records: Olympic and athletic data.
  • Names & Locations: Identifying specific people and places from Wikipedia.

ClashEval is unique because it pairs each question with both a "clean" reference document and multiple "poisoned" versions containing precise perturbations.

Methodology

Our experimental framework utilizes a Perturbation Engine to modify reference documents. For numerical data, we applied scaling factors (e.g., 0.1x to 10x). For categorical data, we created variations ranging from "slight" (e.g., minor name spelling changes) to "comical" (absurd, unrealistic variations).

The ClashEval framework: Testing model arbitration between internal priors and modified context.

We evaluated six top-performing LLMs using three primary metrics:

  1. Accuracy: The frequency with which the model provides the correct answer, regardless of the source.
  2. Prior Bias: How often the model adheres to its own (incorrect) internal answer when the provided context is actually correct.
  3. Context Bias: How often the model adopts incorrect information from the context when its internal knowledge was originally correct.

Key Results and Model Performance

Our benchmarking revealed a significant vulnerability in modern LLMs. Without intervention, models override their own correct internal knowledge in favor of incorrect context over 60% of the time.

Context preference rates across domains. Adherence to context generally decreases as the information becomes more unrealistic (higher absolute log fold change).

The "Unrealistic" Threshold

Models are not blindly obedient. As shown in the results across domains, the more a piece of retrieved information deviates from the truth, the more likely the model is to reject it. For instance, a 10x error in a drug dosage or a "comical" location name is rejected far more frequently than a subtle 1.1x error.

Model Comparisons

  • Claude Opus emerged as the most effective arbitrator, achieving the highest overall accuracy (74.3%) and showing 30% less context bias than GPT-4o.
  • GPT-4o, despite its state-of-the-art status, proved surprisingly gullible, exhibiting high context bias and a tendency to "hallucinate along" with provided errors.
  • Llama-3-8b struggled significantly in these arbitration tasks, trailing with approximately 50% accuracy.

Improving Arbitration via Calibration

We found a strong inverse relationship between a model's token-level confidence in its prior knowledge and its likelihood of being swayed by context. By implementing Calibrated Token Probability Correction—which compares the percentile-calibrated confidence of the prior response against the contextual response—we significantly improved performance. For GPT-4o, this method increased accuracy from 61.5% to 75.4% and reduced context bias from 30.4% to just 10.7%.

Discussion and Limitations

While these findings provide a path toward more reliable RAG systems, several limitations remain. Our study focused on short, fact-based Q&A rather than complex multi-step reasoning. Furthermore, the proposed correction method requires access to token log probabilities, which are not always exposed by commercial API providers. Finally, real-world RAG often involves multiple documents where conflicts may exist between different retrieved sources, rather than just between the model and a single document. Future work will explore these multi-document dynamics and the adversarial risks posed by context bias.

Summary updated

Abstract

Retrieval augmented generation (RAG) is frequently used to mitigate hallucinations and provide up-to-date knowledge for large language models (LLMs). However, given that document retrieval is an imprecise task and sometimes results in erroneous or even harmful content being presented in context, this raises the question of how LLMs handle retrieved information: If the provided content is incorrect, does the model know to ignore it, or does it recapitulate the error? Conversely, when the model's initial response is incorrect, does it always know to use the retrieved information to correct itself, or does it insist on its wrong prior response? To answer this, we curate a dataset of over 1200 questions across six domains (e.g., drug dosages, Olympic records, locations) along with content relevant to answering each question. We further apply precise perturbations to the answers in the content that range from subtle to blatant errors. We benchmark six top-performing LLMs, including GPT-4o, on this dataset and find that LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time. However, the more unrealistic the retrieved content is (i.e. more deviated from truth), the less likely the model is to adopt it. Also, the less confident a model is in its initial response (via measuring token probabilities), the more likely it is to adopt the information in the retrieved content. We exploit this finding and demonstrate simple methods for improving model accuracy where there is conflicting retrieved content. Our results highlight a difficult task and benchmark for LLMs -- namely, their ability to correctly discern when it is wrong in light of correct retrieved content and to reject cases when the provided content is incorrect.
BibTeX
@inproceedings{wu2024how,
  author = {Kevin Wu and Eric Wu and James Zou},
  title = {{How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs' internal prior}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=WGoCZl2itU},
}
Figure