The Stability Fidelity Trade Off in Feature Attribution

26 concepts17 papers1 proposal

When we look at a gradient-based saliency map explaining a deep learning model's prediction, we are often greeted by a chaotic, noisy scattering of pixels. Naturally, our first instinct is to clean it up. By averaging gradients over a local neighborhood or applying input perturbations—techniques like SmoothGrad—we can produce visually clean, stable explanations that look intuitive to humans.

However, this visual cleanup hides a fundamental conflict. In explainable AI (XAI), there is an inescapable mathematical tension between an explanation's stability (how consistent it remains across noise, training runs, or architectures) and its fidelity (how accurately it reflects the model's true internal logic). As a comprehensive benchmarking survey by Giannotti et al. demonstrates, popular post-hoc explainers struggle to simultaneously maximize local accuracy and consistency across multiple runs.

To understand why this trade-off occurs, we have to look at the underlying geometry of deep neural networks.

The Geometry of Jagged Models

Why can't an explanation be both perfectly stable and perfectly faithful? The core reason lies in the non-smooth nature of deep neural networks. Highly parameterized models often learn complex, discontinuous, or "jagged" decision boundaries.

Khan formally proved that the robustness of an explanation is lower-bounded by the smoothness (specifically, the probabilistic Lipschitzness) of the underlying model being explained. If a model's decision logic is genuinely erratic and non-smooth, a faithful explanation must also be erratic and non-smooth to accurately reflect that behavior.

Faithful to a Jagged Model  ──>  Explanation must be Jagged (Low Stability)
         │
         ▼ (Artificially Smooth the Explanation)
         │
Stable & Visually Clean     ──>  Explains a Smoothed Surrogate (Low Fidelity)

When we artificially force an explanation to be stable through input smoothing, we introduce a divergence. As Tan et al. argue, forcing stability on a non-robust model means the resulting explanation no longer describes the actual model; instead, it describes a simplified, smoothed surrogate.

Ye et al. formalized this relationship mathematically for Gaussian smoothing. They showed that as smoothing intensity increases, the resulting saliency map becomes highly stable across different model instances but loses fidelity to the unique local gradient of any single instance. Instead of capturing how this specific model made its decision, the smoothed explanation begins to reflect generalizable features of the broader data distribution.

The Visual Illusion of Clean Heatmaps

In computer vision, this trade-off manifests as a direct conflict between human interpretability and algorithmic honesty. We prefer clean heatmaps that highlight clear objects, but the model might be relying on noisy, pixel-level artifacts.

Dreyer et al. observed that attribution propagation rules providing an honest reflection of a model's decision-making chain frequently produce noisier heatmaps. Conversely, methods that generate highly stable, visually clean explanations often achieve this by acting as simple edge detectors. As Hansen et al. show, these stable methods can become completely unfaithful, generating the same visual heatmaps regardless of the model's internal state.

To bridge this gap without completely sacrificing local structure, Gong et al. introduced a super-pixel-based grouping strategy. By grouping pixels together, they reduce the effective dimensionality of the input space, providing a provable boost to algorithmic stability while attempting to preserve the underlying computational process.

Spectral Analysis and Target Sensitivity

The consequences of smoothing extend beyond visual noise; it also degrades an explanation's ability to distinguish between different classes.

Using spectral analysis (Singular Value Decomposition), Maeß et al. mapped out the mathematical structure of explanation quality. They revealed that explanations are governed by a fundamental trade-off between stability (robustness to local noise) and target sensitivity (class specificity). If you smooth an explanation too aggressively to suppress noise, you inadvertently erase its target sensitivity. The explanation becomes uniform, failing to show what makes the predicted class unique compared to alternative categories.

This loss of specificity is particularly challenging in multi-class scenarios. For instance, Doddaiah et al. found that in deep time-series classifiers, local fidelity often directly conflicts with class-specificity. Without explicit mechanisms to minimize overlapping evidence between classes—such as the mask memory units used in their DEMUX framework—stable explanations quickly lose their diagnostic utility.

Erasing Actionable Recourse and Performance

When smoothing techniques blur out fine-grained gradients to achieve stability, they can also eliminate critical, actionable information.

A stark example of this occurs in algorithmic recourse, where explanations are meant to guide users on how to change their features to alter a model's decision (e.g., "increase your income by \$5,000 to get approved for a loan"). Fokkema et al. mathematically proved that attribution-based explanations cannot be both robust to small input changes and recourse-sensitive for non-linear models.

[Sharp Local Gradient] ───> Provides clear directional signal for Recourse
         │
         ▼ (Apply Smoothing for Robustness)
         │
[Averaged/Flat Signal] ───> Obscures directional changes; Recourse is lost

Because recourse relies on identifying the precise directional path to cross a decision boundary, smoothing the gradients flattens these signals. The smoothing process can result in zero-value saliency maps, effectively blurring the actionable signal into uselessness.

Forcing explanation consistency during a model's training phase can also degrade downstream task performance:

  • Language Models: Xie et al. demonstrated that enforcing consistency constraints on language model attention mechanisms (forcing semantically similar inputs to yield similar rationales) causes a significant drop in downstream task accuracy.
  • Clinical Data: Watson et al. noted a similar performance penalty when training ensemble architectures to output consistent explanations for electronic health records.
  • Optimization Shortcuts: This tension exists because optimizing purely for predictive accuracy often encourages models to take logical shortcuts or leverage background biases. As Ramnath et al. show, when models optimize solely for task performance, their generated explanations frequently suffer from hallucinations and a lack of logical plausibility.

If a model is fundamentally fragile and non-smooth, post-hoc smoothing of its explanations is merely a cosmetic fix that destroys fidelity. Is there a way out?

One solution is to fix the underlying model rather than the explanation. Tan et al. discovered that robust models (such as those trained using adversarial training) naturally yield robust and stable explanations "for free," without any loss of faithfulness. When the underlying decision boundaries are smoothly behaved, the local gradients become inherently stable.

When working with standard, non-robust models, practitioners must carefully design localized constraints rather than relying on global smoothing:

  • Local Sparsity: Lim et al. developed the RelEx framework, which uses $L_1$ regularization within a noisy local neighborhood. This allows the explanation to remain robust against adversarial perturbations while maintaining fidelity to the core features driving the original classification.
  • Selective Minimality: In sequential domains like speech recognition, Wu et al. highlighted a parallel trade-off between minimality and architectural stability. Highly selective explanations that pinpoint the exact causal audio frames tend to vary significantly across different model architectures, whereas larger, less selective explanations remain more stable across systems.

Ultimately, the stability-fidelity trade-off serves as a reminder that an explanation cannot be evaluated in a vacuum. A visually clean, stable heatmap is only valuable if the underlying model has earned it through a smooth, robust decision surface.

Key Foundational Work

Building Reliable Explanations of Unreliable Neural Networks: Locally Smoothing Perspective of Model Interpretation (2021) — contributed 1 concept elements to this direction.
Learning to Explain: Generating Stable Explanations Fast (2021) — contributed 1 concept elements to this direction.
TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robustness_faithfulness_trade_off_d581932c,
  title = {robustness-faithfulness trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robustness-faithfulness-trade-off/rdr_d581932ccf784e5d99bef7593521f27f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_robustness_faithfulness_trade_off_d581932c_v1,
  title = {robustness-faithfulness trade-off},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robustness-faithfulness-trade-off/rdr_d581932ccf784e5d99bef7593521f27f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake