Abstract
This survey provides an in-depth analysis of knowledge conflicts for large language models (LLMs), highlighting the complex challenges they encounter when blending contextual and parametric knowledge. Our focus is on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict. These conflicts can significantly impact the trustworthiness and performance of LLMs, especially in real-world applications where noise and misinformation are common. By categorizing these conflicts, exploring the causes, examining the behaviors of LLMs under such conflicts, and reviewing available solutions, this survey aims to shed light on strategies for improving the robustness of LLMs, thereby serving as a valuable resource for advancing research in this evolving area.
Introduction: The Knowledge Tug-of-War
Large Language Models (LLMs) are trained on massive, static datasets, creating a "parametric memory" of the world. However, as we move toward Retrieval-Augmented Generation (RAG) and real-time AI agents, models frequently encounter "contextual knowledge" that contradicts their internal training. This creates a "tug-of-war" inside the model, leading to hallucinations, sycophancy, and general unreliability.
As illustrated in Figure 1, the survey categorizes these discrepancies into three primary types:
- Context-Memory conflict: When external context (e.g., a 2023 news article) contradicts the model's internal training data (e.g., knowledge from 2022).
- Inter-Context conflict: When multiple provided documents disagree with each other.
- Intra-Memory conflict: When the model provides inconsistent answers to the same question due to variations in phrasing or prompting.
A Framework for Understanding Conflicts
To address the lack of a unified framework, the authors propose a taxonomy that tracks the lifecycle of a knowledge conflict through four distinct stages: Causes, Phenomenon, Analysis of Behaviors, and Solutions.
Causes and Behaviors
Conflicts are often triggered by temporal misalignment (outdated information), misinformation pollution, or bias in the training corpora. When faced with these conflicts, LLMs exhibit several notable behaviors:
- Confirmation Bias: Models tend to favor their internal parametric memory for highly popular entities, even when context suggests otherwise.
- Sycophancy: Models may prioritize "pleasing the user" by adopting incorrect views if the prompt is opinionated.
- Generation-Probe Gap: There is often a significant gap (up to 40%) between what a model "knows" internally and what it actually generates.
Benchmarking and Mitigation
The paper identifies key datasets for evaluating these conflicts, such as PopQA and StrategyQA for Context-Memory issues, and WikiContradiction for Inter-Context discrepancies.
Research shows that misinformation in the context can lead to an 87% drop in performance. While larger models like GPT-4 are better at detecting contradictions (over 70% success rate), smaller models often struggle. Effective mitigation strategies include pre-hoc adjustments (better pre-training) and post-hoc inference-time techniques like Context-aware Decoding (CAD), which can improve adherence to provided facts by over 100%.
Future Frontiers
Despite progress, several challenges remain. Most research relies on artificial conflicts rather than "in the wild" nuances like political disagreements. Future work must also address multimodal conflicts (e.g., text vs. image) and improve the explainability of the internal "knowledge circuits" that resolve these contradictions. Ultimately, models need to move beyond simple rules to dynamically judge the credibility of information sources.
Summary updated