Robustness and Reliability of Post-Hoc Explanations

This direction is merged into another direction: explanation instability (explanation fragility). Its last version stays readable and citable here.

Version 1 of 143 papers47 conceptsfrozen 2026-10-04

Deep neural networks are often described as "black boxes," but the field of Explainable AI (XAI) has developed a suite of tools to peek inside. Saliency maps, feature attribution, and counterfactuals aim to show us why a model made a specific decision. However, a growing body of research reveals a troubling reality: these explanations are often incredibly fragile.

If a model's explanation can be changed by a tiny, human-imperceptible shimmer of noise while the model’s actual prediction stays the same, can we really trust the explanation?

The Fragility of Feature Attribution

The most common way to explain a model is through feature attribution—highlighting which pixels or words were most influential. Methods like Grad-CAM, LIME, and SHAP are the industry standards for this. Yet, as Wojciech Samek and Sebastian Lapuschkin point out in Toward Interpretable Machine Learning, these post-hoc methods are highly susceptible to adversarial perturbations.

The core problem is a decoupling between the model's output and its explanation. In a typical adversarial attack, you change the input to change the prediction. In "Adversarial XAI," as surveyed by Hubert Baniecki and Przemyslaw Biecek in Adversarial Attacks and Defenses in Explainable Artificial Intelligence, the attacker changes the input to change the explanation while keeping the prediction constant.

This fragility isn't just a theoretical curiosity. In security-sensitive domains like Android malware analysis, Ming Fan and colleagues found that interpreters can be easily manipulated to mask malicious behavior, making a compromised model appear benign to an auditor [Can We Trust Your Explanations?].

Why Explanations Break

To understand why explanations are so brittle, we have to look at how they are generated. Most methods rely on the gradient—the slope of the model's decision function.

  1. Gradient Saturation: When a model is very confident, the gradient becomes "flat" (zero). Small changes in the input can cause the gradient to jump wildly, leading to noisy saliency maps.
  2. Off-Manifold Extrapolation: Methods like SHAP or Partial Dependence Plots often calculate importance by "averaging" over data points that don't actually exist. Giles Hooker demonstrates in Why You Should Not Trust Interpretations in Machine Learning that an attacker can place extreme values in these "empty" regions of the data space to surgically manipulate the resulting visual summary.
  3. Lack of End-to-End Stability: Even "Self-Explaining Models" (SEMs) that are designed to be interpretable from the start often only enforce stability on the final prediction, not the internal feature extraction. Haizhong Zheng and Atul Prakash showed that these architectures can still be tricked into providing nonsensical justifications for correct predictions [Analyzing the Interpretability Robustness of Self-Explaining Models].

Building Robust Interpretability

Researchers are now moving beyond simple attribution to methods that are "robust by design." The goal is to ensure that if two inputs are visually identical, their explanations should be identical too.

Smoothing and Decoys

One intuitive fix is to stop looking at a single point and start looking at the neighborhood. Alexander Levine and Soheil Feizi introduced Sparsified SmoothGrad, which provides mathematical "robustness certificates" by averaging gradients over many noisy versions of an input [Certifiably Robust Interpretation in Deep Learning].

Similarly, the DANCE framework, developed by Wenbo Guo and William S. Noble, uses "decoys"—perturbed inputs that the model's internal layers cannot distinguish from the original. By aggregating explanations across these decoys, the method bypasses gradient saturation and produces much cleaner, more stable maps [DANCE: Enhancing saliency maps using decoys].

Geometric and Causal Paths

Other researchers argue that we should respect the "shape" of the data. Standard methods often move in a straight line from a "blank" image to the target image, passing through nonsensical, noisy states. Eslam Zaher proposes Manifold Integrated Gradients (MIG), which follows a curved path along the actual data manifold, leading to explanations that are more resistant to adversarial "hacks" [Manifold Integrated Gradients].

Chao-Han Huck Yang and Pin-Yu Chen suggest moving away from gradients entirely toward Causal Intervention, measuring how much the model's output changes when specific features are masked or modified [When Causal Intervention Meets Adversarial Examples].

The Faithfulness Trade-off

A critical tension exists in this research: the robustness-faithfulness trade-off. As Zeren Tan and Yang Tian argue, if a model is inherently non-robust, a "stable" explanation might actually be lying to you [Robust Explanation for Free or At the Cost of Faithfulness]. If the model is sensitive to noise, the explanation should reflect that noise to be faithful.

True progress in the field, therefore, requires a two-pronged approach: building models that are robust in their decision-making, and developing explanation tools that are mathematically guaranteed to reflect that robustness. As Haochen Liu and colleagues conclude in their survey on Trustworthy AI, interpretability is not a standalone feature—it is a component of a larger system of accountability that must be defended against adversarial manipulation.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robustness_and_reliability_of_post_hoc_e_4d102709,
  title = {Robustness and Reliability of Post-Hoc Explanations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robustness-and-reliability-of-post-hoc-explanations/rdr_4d102709e16b40aeabc4cd57ed94f6b8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_robustness_and_reliability_of_post_hoc_e_4d102709_v1,
  title = {Robustness and Reliability of Post-Hoc Explanations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robustness-and-reliability-of-post-hoc-explanations/rdr_4d102709e16b40aeabc4cd57ed94f6b8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}