Abstract
Providing rationales for decisions can enhance transparency and cultivate trust. Nevertheless, in light of economic incentives and other factors that may encourage manipulation, the reliability of such explanations comes into question. This manuscript introduces a conceptual framework for deceptive explanations and constructs a typology grounded in interdisciplinary literature. We focus on how AI models can generate and detect deceptive explanations. Through a user study of 200 participants, we demonstrate that deceptive explanations can mislead individuals. However, machine learning techniques can discern subtle deceptive tactics with over 80% accuracy, and unsupervised learning can identify inconsistencies when model information is accessible.
Introduction
As AI systems move into critical decision-making roles in banking, healthcare, and law, the demand for "Explainable AI" (XAI) has grown. We seek to understand why a model denied a loan or flagged a medical image. However, this creates a "Liar’s Paradox": what if the explanation itself is a lie?
The core challenge lies in information asymmetry. The "explainer" (the AI developer or the model itself) knows the internal logic, while the "explainee" (the user) does not. This creates incentives for deception, such as "fairwashing"—hiding biased decision-making criteria (like gender or race) behind sanitized, rational-sounding explanations to comply with regulations like the GDPR.
Typology of Deception
Deception in XAI is not merely a technical failure but a deliberate communication strategy. Borrowing from social sciences, the authors categorize deception into three primary mechanisms:
- Scores: Manipulating the relevance or confidence scores assigned to specific features (e.g., overstating, understating, or omitting).
- Presentation: Changing how information is visualized to mislead the user.
- Reasons: Altering the underlying logic, such as equivocating (vague reasons), concealing (omitting real reasons), or fabricating (inventing reasons).
Creation and Detection of Deceptive Explanations
To test these concepts, the study developed automated methods to generate deceptive explanations for text classification using GradCAM as a baseline. Tactics included Omission (removing "forbidden" words from heatmaps) and Noise Addition (distorting relevance scores).
To counter these, two novel unsupervised detection algorithms were proposed:
- ConsistencyChecker: Compares a suspect explanation to a "reference" model. Outliers are flagged as likely deceptive.
- UsefulnessChecker: Tests if a separate classifier can reach the same decision using only the provided explanation. A failure to do so suggests the explanation is a fabrication.
Theoretical Framework and Human Impact
The study proposes a theoretical model to understand how deceptive stimuli affect the "organism" (the user) and their subsequent response.
In a user study of 200 participants using IMDB and Web of Science datasets, humans proved highly vulnerable. On a 5-point scale of agreement with classifications, truthful explanations scored 3.74, while deceptive ones still scored 3.58. This suggests a cognitive bias where users trust a rationale if it confirms the model's prediction, even if the rationale is false.
Key Results and Detection Performance
While humans were easily misled, machine learning detectors showed significant promise:
- Supervised Detection: A CNN trained to spot fakes achieved >80% accuracy, even when deception involved minimal noise (1%).
- Unsupervised Detection: The
ConsistencyCheckereffectively identified blatant deception without requiring labeled training data. - Detection Boundary: The authors proved mathematically that "consistent lying" is undetectable without external domain knowledge. If a model consistently mislabels a "cat" as a "dog," an observer without knowledge of cats cannot detect the deception.
Conclusion
Deceptive XAI represents a significant risk to AI auditing and transparency. While machine learning can detect many forms of manipulation, the "arms race" between liars and detectors will likely intensify. Future work must address multimodal deception and advocate for regulatory "model access" to allow auditors to verify the truthfulness of AI explanations.
Reference: Schneider, J., Meske, C. & Vlachos, M. Deceptive XAI: Typology, Creation and Detection. SN Comput Sci 5, 81 (2024). https://doi.org/10.1007/s42979-023-02401-z
Summary updated