Introduction
Retrieval-Augmented Generation (RAG) is the industry standard for reducing hallucinations by providing Large Language Models (LLMs) with external context. However, models often exhibit a lack of context faithfulness, prioritizing their internal, pre-trained "parametric memory" over the provided text. This creates a tug-of-war when external evidence contradicts what a model already "knows."
This paper investigates the mechanisms behind this stubbornness, revealing that a model's reliance on internal memory is directly tied to its memory strength—the consistency of its answers across paraphrased questions. It also explores how the style of evidence presentation can sway a model's decision.
Methodology
The researchers developed a four-step framework to quantify and test context faithfulness:
- Quantifying Memory Strength: Instead of a single query, the authors use seven paraphrased versions of a question. Memory strength is calculated as the negative entropy of the answer distribution; high consistency indicates high memory strength.
- Generating Conflicts: The framework identifies a model's "Memory Answer" (MA) and generates a "Counter-Memory Answer" (CMA) by substituting entities (e.g., changing a factual number or date).
- Varying Evidence Styles: Four evidence types support the CMA: Direct (simple fact), Repeated (fact stated 2-3 times), Paraphrased (fact in 2-3 different ways), and Indirect (supporting details like dates or titles).
- Evaluation: Models choose between the MA, CMA, or an "Uncertain" option when presented with conflicting evidence.
Key Insights and Results
The study utilized the popQA and Natural Questions (NQ) benchmarks to test models like GPT-4, Claude 3.5, and LLaMA 3.2.
Consistency and Stubbornness
There is a clear correlation between memory strength and internal memory reliance. For questions where GPT-4 exhibited high memory strength on the NQ dataset, it ignored external evidence and relied on its internal memory nearly 50% of the time.
The "Paraphrase Power Move"
A critical finding is that linguistic diversity is more convincing than volume. Simply repeating a fact has limited impact, but presenting the same evidence via multiple paraphrases significantly increases a model's receptiveness. In several tests, paraphrasing was twice as effective as simple repetition at reducing reliance on internal memory.
Model-Specific Behaviors
- Knowledge vs. Faithfulness: Claude 3.5 was found to be highly knowledgeable but also "cautious," frequently choosing the "Uncertain" option rather than stubbornly sticking to its memory when faced with conflict.
- Scale Effects: Larger models like GPT-4 generally have higher memory strength across a broader range of topics, making them harder to "convince" with external context compared to smaller models like LLaMA-3.2-3B.
Conclusion
This research demonstrates that LLM stubbornness is not merely a failure of comprehension but a function of internal memory stability. While paraphrasing provides a powerful tool for improving context faithfulness, current models still struggle with extractive bias and complex reasoning. Future work will focus on extending this framework to "Why" and "How" questions and investigating the disconnect between a model's internal reasoning and its final output selection.
Summary updated