Explainable AI (XAI) is often presented as a solution to the "black box" problem of deep learning, promising to reveal the internal logic of models used in high-stakes domains like finance and law. However, a growing body of research suggests that these explanations can be strategically manipulated. This practice, often termed "fairwashing," allows a model owner to present a biased or unethical model as fair by providing deceptive justifications that mask its true reliance on sensitive attributes.
The Mechanism of Deception
The core of the problem lies in the disconnect between a model's predictive behavior and its post-hoc explanation. As Johannes Schneider et al. argue in Deceptive XAI: Typology, Creation and Detection, explanations can be intentionally crafted to hide illicit decision criteria while appearing perfectly rational to an observer.
One primary way this is achieved is through off-manifold manipulation. In Fairwashing explanations with off-manifold detergent, Christopher J. Anders et al. demonstrate that because real-world data typically resides on a low-dimensional manifold, a model's gradients can be altered in directions orthogonal to this manifold. This allows an adversary to change the model's explanation (which depends on these gradients) without changing its predictions on actual data. The result is a model that behaves unfairly in practice but produces "clean" saliency maps that ignore sensitive features.
Rationalization through Surrogates and Multiplicity
Fairwashing can also take the form of "rationalization," where a complex, biased model is explained using a simpler, fairer-looking surrogate. Ulrich Aïvodji et al. introduced LaundryML in Fairwashing: the risk of rationalization, an algorithm that generates surrogate models that are highly faithful to the original black box's predictions but appear to follow ethical guidelines. The authors further characterized this risk in Characterizing the risk of fairwashing, showing that these deceptive explanations are robust and can even transfer across different models, making them difficult to detect.
This vulnerability is exacerbated by predictive multiplicity. As Charles Marx et al. explain in Predictive Multiplicity in Classification, there are often many different models that achieve the same level of accuracy but rely on different features. This allows a developer to strategically select the model that "looks" the fairest during an audit, even if other equally accurate models would reveal systemic biases.
The Fragility of Interpretability
Even methods designed to be more intuitive, such as counterfactual explanations ("you would have received the loan if your income were $10k higher"), are susceptible. Dylan Slack et al. show in Counterfactual Explanations Can Be Manipulated that model owners can manipulate the decision surface to provide "shortcuts" for preferred subgroups, hiding systemic discrimination behind seemingly helpful suggestions.
The broader field of Adversarial XAI (AdvXAI), surveyed by Hubert Baniecki and Przemyslaw Biecek in Adversarial Attacks and Defenses in Explainable Artificial Intelligence: A Survey, highlights that popular frameworks like SHAP or Grad-CAM are often brittle. Small, imperceptible perturbations to inputs or parameters can lead to wildly different explanations, a point echoed by Matthew Robert Wicker et al. in Robust Explanation Constraints for Neural Networks.
Moving Toward Robust Auditing
The realization that explanations can be "hacked" has led to a shift in how researchers approach AI trust. Tassadit Bouadi et al. argue in Trust in Artificial Intelligence: Beyond Interpretability that transparency alone is insufficient; we need rigorous constraint enforcement and technical robustness.
To counter strategic manipulation, new auditing techniques are emerging:
- Functional Testing: Mohamed Karim Belaid et al. propose Compare-xAI in Compare-xAI: Toward Unifying Functional Testing Methods for Post-hoc XAI Algorithms into a Multi-dimensional Benchmark to evaluate the stability and fidelity of explanations.
- Prior Knowledge: Jade Garcia Bourrée et al. demonstrate in Robust ML Auditing using Prior Knowledge that auditors can detect platforms that mimic fair behavior only during inspection windows by using private, labeled datasets to check for consistency.
- Predictiveness Metrics: The AXE framework, introduced by Kaivalya Rawal et al. in Evaluating Model Explanations without Ground Truth, has shown success in detecting adversarial fairwashing by evaluating explanation quality through its predictive power rather than its appearance.
Ultimately, as Esma Balkir et al. note in Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models, while XAI is a powerful tool for transparency, its susceptibility to manipulation means it cannot be the sole mechanism for accountability. Genuine fairness requires addressing the root causes of bias rather than just polishing the model's justifications.
Key Foundational Work
Fairwashing explanations with off-manifold detergent (2020) — contributed 2 concept elements to this direction.
Fairwashing: the risk of rationalization (2019) — contributed 1 concept elements to this direction.
Predictive Multiplicity in Classification (2020) — contributed 1 concept elements to this direction.