Adversarial Manipulation of Model Explanations

21 concepts15 papers3 proposals

Explainable AI (XAI) is often presented as a solution to the "black box" problem of deep learning, promising to reveal the internal logic of models used in high-stakes domains like finance and law. However, a growing body of research suggests that these explanations can be strategically manipulated. This practice, often termed "fairwashing," allows a model owner to present a biased or unethical model as fair by providing deceptive justifications that mask its true reliance on sensitive attributes.

The Mechanism of Deception

The core of the problem lies in the disconnect between a model's predictive behavior and its post-hoc explanation. As Johannes Schneider et al. argue in Deceptive XAI: Typology, Creation and Detection, explanations can be intentionally crafted to hide illicit decision criteria while appearing perfectly rational to an observer.

One primary way this is achieved is through off-manifold manipulation. In Fairwashing explanations with off-manifold detergent, Christopher J. Anders et al. demonstrate that because real-world data typically resides on a low-dimensional manifold, a model's gradients can be altered in directions orthogonal to this manifold. This allows an adversary to change the model's explanation (which depends on these gradients) without changing its predictions on actual data. The result is a model that behaves unfairly in practice but produces "clean" saliency maps that ignore sensitive features.

Rationalization through Surrogates and Multiplicity

Fairwashing can also take the form of "rationalization," where a complex, biased model is explained using a simpler, fairer-looking surrogate. Ulrich Aïvodji et al. introduced LaundryML in Fairwashing: the risk of rationalization, an algorithm that generates surrogate models that are highly faithful to the original black box's predictions but appear to follow ethical guidelines. The authors further characterized this risk in Characterizing the risk of fairwashing, showing that these deceptive explanations are robust and can even transfer across different models, making them difficult to detect.

This vulnerability is exacerbated by predictive multiplicity. As Charles Marx et al. explain in Predictive Multiplicity in Classification, there are often many different models that achieve the same level of accuracy but rely on different features. This allows a developer to strategically select the model that "looks" the fairest during an audit, even if other equally accurate models would reveal systemic biases.

The Fragility of Interpretability

Even methods designed to be more intuitive, such as counterfactual explanations ("you would have received the loan if your income were $10k higher"), are susceptible. Dylan Slack et al. show in Counterfactual Explanations Can Be Manipulated that model owners can manipulate the decision surface to provide "shortcuts" for preferred subgroups, hiding systemic discrimination behind seemingly helpful suggestions.

The broader field of Adversarial XAI (AdvXAI), surveyed by Hubert Baniecki and Przemyslaw Biecek in Adversarial Attacks and Defenses in Explainable Artificial Intelligence: A Survey, highlights that popular frameworks like SHAP or Grad-CAM are often brittle. Small, imperceptible perturbations to inputs or parameters can lead to wildly different explanations, a point echoed by Matthew Robert Wicker et al. in Robust Explanation Constraints for Neural Networks.

Moving Toward Robust Auditing

The realization that explanations can be "hacked" has led to a shift in how researchers approach AI trust. Tassadit Bouadi et al. argue in Trust in Artificial Intelligence: Beyond Interpretability that transparency alone is insufficient; we need rigorous constraint enforcement and technical robustness.

To counter strategic manipulation, new auditing techniques are emerging:

Ultimately, as Esma Balkir et al. note in Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models, while XAI is a powerful tool for transparency, its susceptibility to manipulation means it cannot be the sole mechanism for accountability. Genuine fairness requires addressing the root causes of bias rather than just polishing the model's justifications.

Key Foundational Work

Fairwashing explanations with off-manifold detergent (2020) — contributed 2 concept elements to this direction.
Fairwashing: the risk of rationalization (2019) — contributed 1 concept elements to this direction.
Predictive Multiplicity in Classification (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fairwashing_f43205b3,
  title = {fairwashing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fairwashing/rdr_f43205b30cda4ca2ba64a58b5dd04487},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_fairwashing_f43205b3_v1,
  title = {fairwashing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fairwashing/rdr_f43205b30cda4ca2ba64a58b5dd04487/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake