The Fragility of Neural Network Explanations

369 concepts245 papers2 proposals

Imagine an AI model trained to detect tumors in medical scans. For a given X-ray, the model correctly identifies a malignancy, and a post-hoc explanation method—like a saliency map—highlights a cluster of pixels directly over the tumor, reassuring the radiologist. Now, imagine adding an imperceptible layer of random static noise to the image. The model’s prediction remains exactly the same: "malignant." However, the saliency map shifts entirely, highlighting a blank patch of healthy tissue instead.

This phenomenon is known as explanation fragility or instability. As deep neural networks are increasingly deployed in safety-critical domains, the reliability of their explanations has become as vital as the reliability of their predictions. As Marta Kwiatkowska et al. emphasize in When to Trust AI: Advances and Challenges for Certification of Neural Networks, an explanation cannot serve as a trustworthy certificate of a model's logic if it fluctuates wildly in response to minor input variations that leave the underlying decision unchanged.

To build reliable interpretability tools, we must understand why post-hoc explanations break down, how we can rigorously measure their instability, and how researchers are engineering solutions to stabilize them.


Why Explanations Break Down

Explanation instability is rarely a random bug; it is an inherent mathematical consequence of how post-hoc explainers interact with the complex geometry of deep neural networks. Three primary mechanisms drive this fragility.

1. Jagged Decision Boundaries and the Smoothness Problem

Many popular explanation methods (such as vanilla gradients or integrated gradients) rely on computing local derivatives to determine feature importance. They ask: If I change this pixel or word by a tiny amount, how much does the output change?

The issue is that neural networks are notorious for having highly non-linear, highly oscillatory decision landscapes. Even if a model makes stable predictions over a large region of the input space, its local gradients can fluctuate chaotically.

Zulqarnain Khan formalized this intuition in Analyzing Explainer Robustness via Probabilistic Lipschitzness of Prediction Functions, proving that the robustness of an explanation is mathematically lower-bounded by the smoothness (specifically, the probabilistic Lipschitzness) of the underlying model. When a model is "jagged," a tiny shift in the input pushes the explainer into a completely different gradient regime, causing feature attributions to flip unpredictably.

2. The Out-of-Distribution Trap

Perturbation-based explainers (like LIME or SHAP) take a different approach: they probe the model by generating a cloud of noisy variants of the input (e.g., masking out words or graying out image patches) and fitting a simple local surrogate model to the outputs.

However, this process frequently violates the data distribution the model was trained on. As Alexander Hepburn demonstrates in Sampling Based On Natural Image Statistics Improves Local Surrogate Explainers, creating arbitrary masks introduces unnatural artifacts and edges that do not exist in real-world data. When forced to evaluate these out-of-distribution samples, the neural network outputs highly unstable, arbitrary predictions, which in turn corrupts the local surrogate model and yields inconsistent heatmaps.

3. Vulnerability to Adversarial Manipulation

Because post-hoc explanations are often decoupled from the global prediction logic, their sensitivity can be maliciously exploited. Adversaries can introduce targeted, imperceptible perturbations designed specifically to fool the explainer while leaving the model's actual prediction intact.

For example, Christopher Burger et al. introduced an adversarial framework called XAIFOOLER in "Are Your Explanations Reliable?" Investigating the Stability of LIME in Explaincribing Text Classifiers by Marrying XAI and Adversarial Attack. They showed that by subtly tweaking a sentence, they could completely erase highly influential features (such as a critical medical symptom) from LIME's top-ranked reasons, proving that un-regularized explainers can easily be tricked into hiding a model's true reasoning.


Diagnosing and Measuring Instability

To fix explanation fragility, the community first needed rigorous ways to quantify it. Early evaluations of interpretability were purely qualitative—researchers simply looked at heatmaps to see if they "made sense" to humans. Recent work has replaced this subjectivity with formal statistical frameworks.

In How Sensitive are Sensitivity-Based Explanations?, Chih-Kuan Yeh et al. introduced two key metrics to evaluate saliency maps:

  • Sensitivity: Measures the maximum change in the explanation map when a tiny, non-essential perturbation is added to the input.
  • Infidelity: Measures how well the explanation tracks meaningful changes in the model's output when important features are removed.

Building on this, Chirag Agarwal et al. argued in Rethinking Stability for Attribution-based Explanations that measuring absolute changes in explanations can be misleading. They proposed Relative Stability, a framework that evaluates whether changes in feature attributions actually correlate with shifts in the model's internal representations, preventing a "false sense of fragility" when an explanation changes for a valid functional reason.


Pathways to Robust Interpretability

Addressing explanation fragility requires moving beyond standard post-hoc methods. Researchers have developed several distinct strategies to enforce stability, ranging from statistical smoothing to designing entirely new model architectures.

Ensembling and Statistical Smoothing

If explanation instability is caused by high-frequency noise in the model's local gradient landscape, a natural solution is to smooth that landscape out. This is the intuition behind methods like SmoothGrad, which averages gradients over a local Gaussian neighborhood around the input. Chih-Kuan Yeh et al. proved that this empirical smoothing mathematically minimizes an explanation's sensitivity.

Taking this statistical defense further, Yuyou Gan proposed MeTFA (Median Test for Feature Attribution) in "Is your explanation stable?": A Robustness Evaluation Framework for Feature Attribution. Instead of relying on a single explanation or a simple mean, MeTFA applies non-parametric hypothesis testing to the median of a sampled distribution of explanations, providing statistical guarantees that filter out random noise and adversarial perturbations.

Smoothing the Model via Adversarial Training

Rather than trying to smooth a jagged model during the explanation phase, why not make the model itself smooth during training?

Xi Wu demonstrated this synergy in Concise Explanations of Neural Networks using Adversarial Training. By training neural networks to resist $\ell_\infty$-bounded adversarial attacks, the model is forced to flatten its local decision boundaries. A beautiful side effect of this robust training is that the model's gradients become inherently stable and sparse, naturally yielding clean, robust explanations without requiring any post-hoc smoothing overhead.

Causal and Counterfactual Interventions

To bypass the fragility of single-point gradients and correlational shortcuts altogether, some researchers have turned to causal inference.

In Causal analysis for robust interpretability of neural networks, Ola Ahmad et al. developed a framework that performs "path interventions" directly on the internal layers of a network. By systematically flipping internal neural switches to observe downstream effects, they construct causal explanatory graphs that reflect the true, stable pathways of a model's logic, ignoring superficial input noise.

Similarly, Xuexin Chen et al. introduced FANS (Feature Attribution with Necessity and Sufficiency) in Feature Attribution with Necessity and Sufficiency via Dual-stage Perturbation Test for Causal Explanation. FANS uses a dual-stage counterfactual framework to explicitly separate features that are truly necessary and sufficient for a prediction from local artifacts of the decision boundary, ensuring the explanation remains grounded in core logic.

Self-Interpretable Architectures

The most radical solution to post-hoc fragility is to abandon post-hoc explainers entirely, designing models that are inherently interpretable by construction.

For tabular data, Arlind Kadra et al. introduced Interpretable Mesomorphic Networks (IMNs) in Interpretable Mesomorphic Networks for Tabular Data. IMNs use a deep hypernetwork to dynamically generate instance-specific linear weights. Because the model's prediction is a linear combination of its features at that exact point, it provides authentic, mathematically stable feature importances instantly, bypassing post-hoc approximations entirely.

For computer vision, Yipei Wang proposed SITE (Self-Interpretable model with Transformation Equivariant Interpretation) in Self-Interpretable Model with Transformation Equivariant Interpretation. SITE enforces a structural constraint called transformation equivariance. If you rotate, scale, or shift an input image, the model's internal explanation mechanism is mathematically forced to rotate, scale, or shift in the exact same way. This structural guarantee prevents the chaotic re-allocation of feature importance that plagues traditional post-hoc saliency maps under geometric shifts.

Strategy Core Mechanism Primary Benefit Key Reference
Statistical Smoothing Averages gradients or attributions over a local neighborhood. Removes high-frequency noise from post-hoc maps. Yeh et al., Gan
Adversarial Training Flattens decision boundaries via robust optimization. Inherently stabilizes gradients at the model level. Wu
Causal Inference Uses path interventions and counterfactual testing. Isolates true logical necessity, ignoring boundary artifacts. Ahmad et al., Chen et al.
Self-Interpretable Design Constrains model architecture (e.g., equivariance, hypernetworks). Eliminates post-hoc approximations entirely; guarantees consistency. Kadra et al., Wang

Summary

Explanation fragility highlights a critical gap between a model's global classification performance and its local mathematical behavior. When an explanation tool relies on a highly non-linear, unregularized decision surface, it captures transient local artifacts rather than robust semantic logic.

By diagnosing these failures through metrics like sensitivity and relative stability, the field has shifted toward solutions that smooth the underlying model landscape, isolate causal pathways, or enforce structural constraints directly within the network architecture. Ultimately, ensuring that an AI can explain its decisions consistently across minor, non-semantic variations is a prerequisite for deploying trustworthy systems in the real world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_instability_explanation_frag_ba116733,
  title = {explanation instability (explanation fragility)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-instability-explanation-fragility/rdr_ba11673308954e7f85f9a1d3c5fd9e85},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_explanation_instability_explanation_frag_ba116733_v1,
  title = {explanation instability (explanation fragility)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-instability-explanation-fragility/rdr_ba11673308954e7f85f9a1d3c5fd9e85/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake