Explainable Artificial Intelligence (XAI) is traditionally championed as a mechanism to build trust and help human operators calibrate when to accept or reject machine recommendations. However, a growing body of empirical research reveals a troubling paradox: instead of fostering healthy skepticism, system explanations often act as tools of superficial persuasion. When an AI system outputs an incorrect prediction, the presence of an accompanying explanation frequently induces explanation-induced over-reliance, causing human users to accept errors they might have otherwise caught.
To understand why this phenomenon occurs, we must look beyond the technical accuracy of explanations and examine how they interact with human cognitive heuristics.
The Plausibility Trap and Superficial Cues
When users evaluate an AI's output, they rarely audit the underlying mathematical logic. Instead, they rely on cognitive shortcuts—superficial cues that serve as proxies for competence.
One primary cue is linguistic fluency. In a clinical decision-support study, Maxime Kayser et al. demonstrated in Their paper, Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting, that highly persuasive natural language justifications humanize AI systems. This humanization masks underlying inaccuracies by making the system's reasoning appear rational and authoritative. Similarly, M. Ramesh et al. found in Assessing WildfireGPT: a comparative analysis of AI models for quantitative wildfire spread prediction that conversational fluency creates a false impression of technical mastery, leading users to trust qualitative explanations even for complex quantitative forecasting tasks where the model is fundamentally weak.
Another powerful heuristic is explanation length and detail. As Mark Steyvers et al. observe in What large language models know and what people think they know, human users frequently treat the sheer volume of an explanation as a proxy for expertise. Long-winded, detailed rationales effectively mask low model confidence, tricking users into accepting erroneous outputs because the justification sounds thorough.
This creates what Mohamed Karim Belaid et al. describe in Compare-xAI: Toward Unifying Functional Testing Methods for Post-hoc XAI Algorithms into a Multi-dimensional Benchmark as an illusion of explanatory depth. Users overestimate their understanding of the system's internal logic simply because the explanation is structured beautifully, leading to misplaced trust in unfaithful or volatile models.
Rationalization versus Verifiability
A critical reason why explanations fail to prevent errors is the structural difference between rationalization and verifiability.
Current XAI methods typically act as rationalizers. As Gagan Bansal et al. argue in Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance, these methods focus on making a prediction seem plausible rather than providing the necessary information for a human to verify its accuracy. When an AI system rationalizes a mistake, it constructs a coherent narrative around a false conclusion.
In his paper In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making, Raymond Fok explains that explanations only improve human-AI collaboration when they offer a "shortcut" to independent verification. Without this verifiability, users are forced to either blindly trust the plausible narrative or expend significant cognitive effort to audit the claim from scratch. In practice, users default to trust.
[AI Prediction + Rationalizing Explanation] ──> Triggers Heuristic Processing ──> Over-reliance
[AI Prediction + Verifiable Evidence] ──> Enables Independent Audit ──> Appropriate Reliance
This problem is compounded by the fact that even faithful explanations—those that accurately reflect the model's internal logic—can be deceptive. Anubrata Das et al. showed in ProtoTEx: Explaining Model Decisions with Prototype Tensors that when a model's internal logic is transparent but fundamentally flawed, the logical structure of the explanation still induces over-trust. The user sees a clear, step-by-step chain of reasoning and assumes the conclusion must be correct, failing to realize that the premise or the weights are broken.
The Placebo Effect of Interpretability
In many human-AI interaction scenarios, explanations function less as informative tools and more as psychological placebos.
Chenhao Tan et al. demonstrated this effect in On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection, discovering that the mere presence of technical-looking explanations increases human trust in a machine's output, even if the explanations are generated completely at random. The aesthetic presentation of data—such as highlighted features or charts—signals competence, short-circuiting critical evaluation.
This placebo effect creates a dangerous decoupling between user confidence and actual accuracy. As Kasun Amarasinghe et al. report in On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods, providing explanations can significantly boost a user's subjective confidence and perceived information quality without yielding any objective improvement in decision-making performance. The user feels more capable and secure, yet they are making the same number of errors—they are simply more confident in their mistakes.
Fostering Appropriate Reliance
To mitigate explanation-induced over-reliance, the focus of XAI is shifting from maximizing user trust to engineering appropriate reliance and healthy skepticism.
The authors of Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation, Roel Visser et al., emphasize that fostering calibrated distrust is just as critical as building trust. If an explanation appears too slick or intelligent, it must be balanced with design interventions that force cognitive friction.
Several strategies have emerged to break the spell of persuasive explanations:
- Withholding Labels: Jiechen Xu et al. found in On the Role of Large Language Models in Crowdsourcing Misinformation Assessment that providing explanations without showing the final AI prediction label forces users to use the explanation to evaluate the problem critically, rather than using it as a rubber stamp for the AI's conclusion.
- Multimodal Complementarity: Combining different explanation modalities can disrupt heuristic thinking. For instance, Maxime Kayser et al. noted that pairing highly fluent text justifications with visual saliency maps helps clinicians maintain critical distance, as the visual maps can expose when the fluent text is hallucinating or anchoring on irrelevant features.
- Active Verification Frameworks: Providing background context or source provenance directly within the interface allows users to perform the independent validation required for true complementary performance, rather than relying on the AI's self-serving rationalizations.