The primary goal of Explainable AI (XAI) is to open the "black box," providing transparency that allows humans to verify, trust, and collaborate with machine learning models. However, a growing body of research suggests that explanations often achieve the opposite: they act as a "persuasive sales pitch" that induces blind trust, masks model failures, and encourages humans to ignore their own better judgment.
This phenomenon, often called the Persuasion Paradox, occurs when the clarity or fluency of an explanation becomes a proxy for the accuracy of the underlying prediction.
The Sales Pitch Effect
In an ideal human-AI team, an explanation should help a person catch an AI's mistake. But as Gagan Bansal and colleagues found in Does the Whole Exceed its Parts?, explanations often lead to "blind trust." Instead of serving as an objective diagnostic tool, explanations function as a persuasive justification.
This is particularly acute with Large Language Models (LLMs). Judith Sieker and her team describe an "illusion of competence" where fluent, natural language justifications lead users to overestimate an AI’s abilities. Even when a system is fundamentally broken—referencing visual features it cannot actually perceive—the linguistic quality of the explanation masks the failure. Xuhai Xu notes that this "narrative persuasion" is often more dangerous than traditional heatmaps because it induces a false sense of understanding in non-experts.
Automation Bias and the Cognitive Crutch
When an AI provides a plausible-sounding reason for its choice, humans are prone to automation bias—the tendency to uncritically follow automated suggestions. In clinical settings, this can be a "double-edged sword." Susanne Gaube and Maxime Kayser have shown that while explanations help non-specialists, they can lead experts to accept incorrect advice because the justification appears superficially convincing.
This happens because explanations often trigger confirmation bias. As the HIVE framework researchers demonstrate, users frequently find justifications for incorrect predictions to be convincing if they align with any vague intuition. Hubert Baniecki and Przemyslaw Biecek even found that humans will rationalize nonsensical visual highlights—like an AI using car parts to identify a bird—as long as the model points to something that looks vaguely relevant.
Fairwashing and Manufactured Legitimacy
The risks of the Persuasion Paradox extend into ethics and law. Maria De-Arteaga and Jakob Schoeffer describe a "fairwashing" effect, where professional-looking explanations increase a user's perception of procedural fairness without actually reflecting the model's underlying biases.
By providing a "placebo" of transparency, these systems can manufacture legitimacy for biased or discriminatory decisions. Hofit Wasserman Rozen argues that this fundamentally misaligns with the legal "right to explanation," as technical justifications often prioritize user trust over the ability to meaningfully contest a decision. Instead of empowering the user, the explanation becomes a tool for manipulation.
Beyond Narratives: The Path to Verifiability
If explanations are often just "math-washing" or "placebic information," how do we build truly useful XAI? The research points toward two key shifts:
- Prioritizing Verifiability: Raymond Fok argues that explanations only provide value when they enable a human to independently and quickly confirm the AI's recommendation. Without a "shortcut" to the truth, the human is just guessing whether the AI's story is right.
- Focusing on Raw Evidence: Greta Warren and colleagues found that users often benefit more from access to raw underlying evidence than from the AI's natural language summary. Similarly, Andreas Theissler suggests that "white-box" assistance, which promotes active verification rather than passive consumption, can boost expert performance by up to 30%.
Ultimately, as David Bau and Leilani H. Gilpin suggest, the field must balance interpretability (human understanding) with completeness (mechanical accuracy). If we optimize only for the former, we risk creating systems that are perfectly understandable, highly persuasive, and confidently wrong.