Fooling SHAP with Stealthily Biased Sampling

gabriel laberge, Ulrich Aïvodji, Satoshi Hara, Mario Marchand, Foutse KhomhICLR 2023 · OpenReview

TL;DR

SHAP (Shapley Additive Explanations) is the industry standard for auditing model fairness, but it has a hidden vulnerability. This paper demonstrates a novel attack that manipulates SHAP explanations by stealthily re-weighting the background data used for sampling. This allows an adversary to hide model bias by up to 90% without ever touching the underlying model itself.

The Illusion of Explainability

As machine learning is increasingly deployed in high-stakes areas like finance and justice, "explainability" has become a strict requirement for establishing trust. SHAP is widely used to identify which features—such as race or gender—contribute most to a model's decision. Auditors rely on these attributions to ensure models aren't leaning on sensitive attributes.

Previous research showed that you could "fool" SHAP by modifying the model's behavior on out-of-distribution data. However, those attacks are often detectable if an auditor checks the model's consistency. This paper asks a more dangerous question: Can an adversary manipulate the explanation while leaving the model completely intact?

The authors identify a critical vulnerability in how SHAP estimates expectations. Because SHAP compares a specific input against a "background distribution" of data, an adversary who controls that background dataset can "cherry-pick" samples to make the model appear fairer than it actually is.

The "Fool SHAP" Attack

The authors introduce an attack called Fool SHAP, which focuses on manipulating Global Shapley Values (GSV)—the average importance of a feature across a whole group. The attack unfolds in four main steps:

  1. Stealthily Biased Sampling: Instead of uniform sampling from the background data, the attacker assigns non-uniform weights to existing data points.
  2. Optimization via Minimum Cost Flow (MCF): The attacker wants to find weights that minimize the attribution of a sensitive feature. The authors frame this as a Minimum Cost Flow problem, which can be solved efficiently in polynomial time.
  3. The Stealth Constraint: To avoid detection, the manipulated distribution must remain "close" to the original data. The authors use the Wasserstein distance (also known as the Earth Mover's Distance) to ensure the fake background distribution is statistically indistinguishable from the real one.
  4. Detection Evasion: The attack is designed to bypass standard auditing tools like the Kolmogorov-Smirnov (KS) and Wald tests, which auditors use to verify if a provided benchmark dataset is legitimate.

Key Insights

  • The Model is Not the Only Attack Surface: Most explainability research focuses on the model weights. This paper proves that the data used to generate the explanation is just as vulnerable.
  • Reproducibility as a Trap: Because the attacker provides a "benchmark dataset" to the auditor, the auditor will run the SHAP library on that data and get the exact (manipulated) results the attacker intended. This creates a false sense of "verifiable" fairness.
  • Sampling is a Weak Link: SHAP’s reliance on Monte-Carlo estimates from a background set is a fundamental architectural vulnerability that can be exploited by anyone providing "audit-ready" datasets.

Key Results

The researchers evaluated their attack on several datasets, including COMPAS (criminal recidivism) and Adult Income (census data).

  • Massive Attribution Reduction: In the COMPAS and Marketing datasets, the attack achieved a 90% relative decrease in the importance of the sensitive feature.
  • Total Stealth: In all experiments, the "detection rate" by auditors remained below the 5% significance threshold. Essentially, the auditors' statistical tests performed no better than random guessing at identifying the fraud.
  • Real-World Impact: On the Adult Income dataset, the attack successfully moved the gender feature from the most negative attribution to the 6th, effectively burying the evidence of model bias.
Comparison of original and manipulated SHAP values on the Adult Income dataset.

Limitations and Future Work

  • Interventional vs. Observational: The current attack applies to "interventional" SHAP (the default in the SHAP library), which ignores feature correlations. Future work should explore "observational" SHAP.
  • Black-Box Access: The attacker needs access to the model's outputs to calculate the optimal weights, though they don't need to see the model's internal parameters.
  • Broader Applications: The authors suggest this sampling-based attack could potentially be adapted to other explanation methods beyond SHAP, such as LIME or Integrated Gradients.

Summary updated

Abstract

SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus a background distribution. Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired explanations. However, existing attacks focus solely on altering the black-box model itself. In this paper, we propose a complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution. In the context of fairness audit, we show that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups while remaining undetected. More precisely, experiments performed on real-world datasets showed that our attack could yield up to a 90\% relative decrease in amplitude of the sensitive feature attribution. These results highlight the manipulability of SHAP explanations and encourage auditors to treat them with skepticism.
BibTeX
@inproceedings{laberge2023fooling,
  author = {gabriel laberge and Ulrich Aïvodji and Satoshi Hara and Mario Marchand and Foutse Khomh},
  title = {{Fooling SHAP with Stealthily Biased Sampling}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=J4mJjotSauh},
}
Figure