Adversarial Manipulation of Post-Hoc Interpretability

59 concepts42 papers

Explainable AI (XAI) was designed to solve the "black box" problem, providing transparency in high-stakes domains like medical diagnosis and legal sentencing. By using methods like saliency maps or feature attribution, researchers aimed to show why a model made a specific decision. However, a growing body of research reveals that these explanations are often "fragile"—they can be systematically manipulated to show whatever an attacker desires without changing the model's actual prediction.

The Fragility of Feature Attribution

The core vulnerability of post-hoc explanation methods is that they are often not robust to small, imperceptible changes in input data. Ghorbani et al. et al. demonstrated in Interpretation of Neural Networks Is Fragile that an adversary can redirect a user's attention to completely irrelevant features. For instance, a small amount of noise added to an X-ray could leave the "cancer" diagnosis unchanged while causing the saliency map to highlight a random, healthy patch of tissue instead of the actual lesion.

This phenomenon occurs because the "explanation surface"—the landscape of gradients used to generate heatmaps—is much more complex and curved than the model's decision boundary. Dombrowski et al. et al. argued in Explanations can be manipulated and geometry is to blame that this high curvature allows attackers to optimize a loss function that minimizes the distance between a current gradient and a target visual pattern (like a specific word or shape). Because the model's output remains constant during this optimization, the resulting explanation becomes a "lie" that no longer reflects the underlying classification logic.

Model Scaffolding and Deceptive Fine-Tuning

Manipulation is not limited to input perturbations; the model itself can be designed to deceive. Taher describes a technique called scaffolding, where an attacker wraps a biased or malicious model in an external layer. This layer is specifically engineered to produce benign explanations for a human analyst while the internal "black box" continues to operate on biased logic.

Similarly, Dimanov showed in You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods that adversarial fine-tuning can modify a pre-trained model to suppress the importance of sensitive variables like race or gender. By suppressing local gradients related to these features without altering global behavior, a model can appear "fair" to tools like LIME or SHAP while still relying on discriminatory data to make its predictions.

Attacking the Audit Trail

Even the tools used to audit models over time are susceptible. Baniecki et al. et al. found that Partial Dependence (PD) curves—popular for explaining how specific features influence predictions across a dataset—can be "bent" through data poisoning. In Fooling Partial Dependence via Data Poisoning, the authors show that injecting a few poisoned samples into the auditing dataset can hide model biases or manufacture desired trends, effectively tricking regulators.

This vulnerability extends to even the simplest models. Li et al. et al. demonstrated in Optimal Feature Manipulation Attacks Against Linear Regression that an adversary can flip the perceived importance of variables in a linear regression by injecting just a single poisoned data point. This suggests that interpretability is a distinct security surface that requires its own specialized defenses.

Toward Robust Interpretability

To counter these vulnerabilities, researchers are moving toward "adversarial-grade" explanation stability. Several strategies have emerged:

  • Geometric Smoothing: Dombrowski et al. et al. suggested that "smoothing" the network's geometry can mitigate the curvature that attackers exploit.
  • Latent Space Recourse: In the context of textual explanations, El Zini found that generating counterfactuals in a model's latent representation is significantly more robust than searching through discrete text, as it prevents minor perturbations from degrading the explanation's realism.
  • Stability Regularizers: Chen et al. et al. proposed the R2ET regularizer in Training for Stable Explanation for Free, which penalizes models when their top-$k$ important features shift due to noise, providing stability at a low computational cost.
  • Causal Analysis: Jha et al. et al. argued for a "deliberative" layer that performs causal analysis to identify influential features, noting that adversarial examples are often more fragile than benign ones when subjected to feature masking.

As Fan et al. et al. concluded in their sanity checks for malware analysis, the current fragility of interpreters means they cannot yet be fully trusted in adversarial environments. The field is shifting from simply generating explanations to verifying their faithfulness and robustness, ensuring that the "why" behind an AI's decision cannot be easily fabricated.

Key Foundational Work

Interpretation of Neural Networks Is Fragile (2019) — contributed 2 concept elements to this direction.
Why the Failure? How Adversarial Examples Can Provide Insights for Interpretable Machine Learning (2018) — contributed 1 concept elements to this direction.
Explaining Black-box Android Malware Detection (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_explanation_manipulation_846d39d4,
  title = {adversarial explanation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-explanation-manipulation/rdr_846d39d43e0f4779b686c493b83017d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_adversarial_explanation_manipulation_846d39d4_v1,
  title = {adversarial explanation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-explanation-manipulation/rdr_846d39d43e0f4779b686c493b83017d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}

See something wrong? Report a mistake