Abstract
Large language models (LLMs) acquire extensive knowledge during pre-training, known as their parametric knowledge. However, in order to remain up-to-date and align with human instructions, LLMs inevitably require external knowledge during their interactions with users. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge? To investigate this question, we propose a framework that systematically elicits LLM parametric knowledge and introduces external knowledge. Specifically, we uncover the impacts by constructing a parametric knowledge graph to reveal the different knowledge structures of LLMs, and introduce external knowledge through distractors of varying degrees, methods, positions, and formats. Our experiments on both black-box and open-source models demonstrate that LLMs tend to produce responses that deviate from their parametric knowledge, particularly when they encounter direct conflicts or confounding changes of information within detailed contexts. We also find that while LLMs are sensitive to the veracity of external knowledge, they can still be distracted by unrelated information. These findings highlight the risk of hallucination when integrating external knowledge, even indirectly, during interactions with current LLMs.
Introduction: The Ripple Effect
Large Language Models (LLMs) effectively "memorize" a world of facts during pre-training, a phenomenon known as parametric knowledge. In practical applications like Retrieval-Augmented Generation (RAG), these models are often provided with external context to ensure they stay current. However, a significant challenge arises when this external information conflicts with the model's internal beliefs.
While previous research has focused on direct contradictions, this paper explores the "ripple effect": how a distractor at one step of a multi-hop reasoning chain can derail the entire logic pathway. For instance, if a model is told "The capital of China is London," it may logically follow that the tallest building in China's capital is The Shard, despite "knowing" otherwise internally.
Methodology: The PKG Framework
The authors propose a systematic framework to interrogate LLMs and measure the impact of external noise.
Parametric Knowledge Graph (PKG) Construction
Instead of comparing model outputs to real-world ground truth, the researchers construct a Parametric Knowledge Graph (PKG) based on the model's actual internal beliefs. By using 63 relationship rules (e.g., "capital of," "born in"), they extract entities and their connections directly from the model. This ensures that any deviation observed is a true "merge conflict" with the model's own data.
The Distractor Framework
The researchers introduce external knowledge designed to interfere with the PKG across four key dimensions:
- Methods: Altering the Object (the answer), the Subject (the entity), or using Indirect distractors.
- Degrees: "Type Match" (replacing a city with another city) vs. "Type Shift" (replacing a city with an unrelated category like an animal).
- Positions: Placing the distractor at various "hops" in the reasoning chain.
- Formats: Presenting the distractor as a simple sentence versus a persuasive, lengthier paragraph.
Experimental Results and Insights
The study involved constructing 8 unique PKGs for models like GPT-3.5 and MPT-7B, resulting in over 17,000 query rounds.
The Confidence Paradox
A key finding is the Confidence Paradox: while models are more likely to stick to internal knowledge when they are highly confident, if they do follow a distractor, their confidence in the incorrect answer often increases. The external information acts as a "bolster," making the model more certain of its error.
Persuasion and Plausibility
LLMs exhibit a strong bias toward lengthier, more detailed context. A distractor hidden in a 3-4 sentence paragraph is significantly more effective at misleading a model than a single-sentence distractor. Furthermore, models struggle with "Type Match" distractors because the replacement is semantically plausible, whereas they easily reject "Type Shift" nonsense.
Model Vulnerability
Consistency scores—the model's ability to stick to its internal truth—dropped significantly under interference. GPT-3.5's consistency fell to 42.9% in certain multi-hop scenarios, while the open-source MPT-7B was even more fragile, often falling below 35%.
Conclusion
This research underscores the inherent risks of hallucination when integrating external knowledge into LLMs. Even indirect interference can cause a model to abandon its internal logic. The authors suggest that future development should focus on "verification layers" that can detect and mitigate these persuasive hallucinations by comparing model behavior across different context lengths.
Summary updated