Deceptive XAI: Typology, Creation and Detection

Johannes Schneider, Christian Meske, Michalis VlachosSN Comput. Sci 2024 · OpenReview

Abstract

Providing rationales for decisions can enhance transparency and cultivate trust. Nevertheless, in light of economic incentives and other factors that may encourage manipulation, the reliability of such explanations comes into question. This manuscript introduces a conceptual framework for deceptive explanations and constructs a typology grounded in interdisciplinary literature. We focus on how AI models can generate and detect deceptive explanations. Through a user study of 200 participants, we demonstrate that deceptive explanations can mislead individuals. However, machine learning techniques can discern subtle deceptive tactics with over 80% accuracy, and unsupervised learning can identify inconsistencies when model information is accessible.

Introduction

As AI systems move into critical decision-making roles in banking, healthcare, and law, the demand for "Explainable AI" (XAI) has grown. We seek to understand why a model denied a loan or flagged a medical image. However, this creates a "Liar’s Paradox": what if the explanation itself is a lie?

The core challenge lies in information asymmetry. The "explainer" (the AI developer or the model itself) knows the internal logic, while the "explainee" (the user) does not. This creates incentives for deception, such as "fairwashing"—hiding biased decision-making criteria (like gender or race) behind sanitized, rational-sounding explanations to comply with regulations like the GDPR.

Typology of Deception

Deception in XAI is not merely a technical failure but a deliberate communication strategy. Borrowing from social sciences, the authors categorize deception into three primary mechanisms:

  1. Scores: Manipulating the relevance or confidence scores assigned to specific features (e.g., overstating, understating, or omitting).
  2. Presentation: Changing how information is visualized to mislead the user.
  3. Reasons: Altering the underlying logic, such as equivocating (vague reasons), concealing (omitting real reasons), or fabricating (inventing reasons).
Typology of Deceptive Explanations

Creation and Detection of Deceptive Explanations

To test these concepts, the study developed automated methods to generate deceptive explanations for text classification using GradCAM as a baseline. Tactics included Omission (removing "forbidden" words from heatmaps) and Noise Addition (distorting relevance scores).

To counter these, two novel unsupervised detection algorithms were proposed:

  • ConsistencyChecker: Compares a suspect explanation to a "reference" model. Outliers are flagged as likely deceptive.
  • UsefulnessChecker: Tests if a separate classifier can reach the same decision using only the provided explanation. A failure to do so suggests the explanation is a fabrication.

Theoretical Framework and Human Impact

The study proposes a theoretical model to understand how deceptive stimuli affect the "organism" (the user) and their subsequent response.

Theoretical Model of Deceptive Explanations

In a user study of 200 participants using IMDB and Web of Science datasets, humans proved highly vulnerable. On a 5-point scale of agreement with classifications, truthful explanations scored 3.74, while deceptive ones still scored 3.58. This suggests a cognitive bias where users trust a rationale if it confirms the model's prediction, even if the rationale is false.

Key Results and Detection Performance

While humans were easily misled, machine learning detectors showed significant promise:

  • Supervised Detection: A CNN trained to spot fakes achieved >80% accuracy, even when deception involved minimal noise (1%).
  • Unsupervised Detection: The ConsistencyChecker effectively identified blatant deception without requiring labeled training data.
  • Detection Boundary: The authors proved mathematically that "consistent lying" is undetectable without external domain knowledge. If a model consistently mislabels a "cat" as a "dog," an observer without knowledge of cats cannot detect the deception.

Conclusion

Deceptive XAI represents a significant risk to AI auditing and transparency. While machine learning can detect many forms of manipulation, the "arms race" between liars and detectors will likely intensify. Future work must address multimodal deception and advocate for regulatory "model access" to allow auditors to verify the truthfulness of AI explanations.

Reference: Schneider, J., Meske, C. & Vlachos, M. Deceptive XAI: Typology, Creation and Detection. SN Comput Sci 5, 81 (2024). https://doi.org/10.1007/s42979-023-02401-z

Summary updated

Abstract

Providing rationales for decisions can enhance transparency and cultivate trust. Nevertheless, in light of economic incentives and other factors that may encourage manipulation, the reliability of such explanations comes into question. This manuscript builds upon a previous conference paper\(^*\) by introducing a conceptual framework for deceptive explanations and constructing a typology grounded in interdisciplinary literature. The focus of our work is on how AI models can generate and detect deceptive explanations. In our empirical evaluation, we focus on text classification and introduce modifications to the explanations generated by GradCAM, a well-established method for explaining neural networks. Through a user study comprising 200 participants, we demonstrate that these deceptive explanations have the potential to mislead individuals. However, we also demonstrate that machine learning (ML) techniques can discern even subtle deceptive tactics with an accuracy exceeding 80%, given sufficient domain expertise. Furthermore, even in the absence of domain knowledge, unsupervised learning can be employed to identify inconsistencies in the explanations, provided that fundamental information about the underlying predictive model is accessible.
BibTeX
@inproceedings{schneider2024deceptive,
  author = {Johannes Schneider and Christian Meske and Michalis Vlachos},
  title = {{Deceptive XAI: Typology, Creation and Detection}},
  booktitle = {SN Comput. Sci},
  year = {2024},
  url = {https://openreview.net/forum?id=aXli78qZ5U},
}
Figure