Model Parameter Invariance in Visual Explanations

29 concepts19 papers1 proposal

When looking at a visual saliency map—a heatmap highlighting the pixels that supposedly drove a deep neural network's prediction—it is easy to be impressed by its precision. If a model classifies an image as a "Siberian Husky" and the saliency map perfectly outlines the dog's ears and snout, our human intuition jumps to a comforting conclusion: the model has learned a robust, human-like representation of a husky.

This is the visual fidelity illusion. We mistake clean, highly detailed edge detection for a faithful reflection of a model's internal decision logic. In reality, a major crisis in explainable AI (XAI) has revealed that many popular attribution methods generate nearly identical heatmaps regardless of whether the model's weights are meticulously trained or completely randomized. They are explaining the input image, not the model.

The Model Parameter Randomization Test

To expose this decoupling between explanations and model logic, Adebayo et al. introduced a foundational sanity check known as the Model Parameter Randomization Test (MPRT) in Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values and Sanity Checks for Saliency Maps.

The core intuition of a parameter sanity check mirrors a basic control experiment: if an explanation truly describes how a specific model transforms inputs into outputs, then changing the model's internal parameters must change the explanation.

[Input Image] ---> [ Fully Trained Model ] ---> [ Saliency Map A ]
                                                       |
                                            Are these identical?
                                                       |
[Input Image] ---> [ Randomized Weights  ] ---> [ Saliency Map B ]

In an MPRT, researchers randomize the weights of a neural network—either all at once or in a cascading fashion from the top output layers down to the bottom input layers—and regenerate the saliency maps.

The results were startling: methods like Guided Backpropagation and Guided Grad-CAM produced visually crisp, virtually unchanged heatmaps even when explaining a network with completely scrambled, random weights. Because these explanations are invariant to the model's parameters, they fail to provide any insight into the specific model they are meant to describe. Instead, they function as sophisticated, input-driven edge detectors.

This failure mode is not unique to standard image datasets; it spans critical domains. In medical image analysis, Kuijf et al. noted that visual saliency methods frequently act as simple image filters rather than indicators of diagnostic logic. Similarly, in high-stakes fields like neuroimaging (Munroe et al.) and cosmological weak lensing (Zorrilla Matilla et al.), unvalidated saliency maps risk providing misleading explanations that capture structural properties of the data rather than learned scientific insights.

Why Modified Backpropagation Fails

Why do these attribution methods decouple from the model's parameters? The answer lies in the mathematics of how gradients are modified during backward passes.

In When Explanations Lie: Why Many Modified BP Attributions Fail, Sixt et al. mathematically demonstrated that many modified backpropagation algorithms (including Layer-wise Relevance Propagation and Guided Backprop) can be formalised as a chain of non-negative matrices. As the explanation propagates backward through deep layers, this matrix multiplication converges to a rank-1 state.

When a system converges to a rank-1 state, the direction of the output vector becomes entirely dominated by the structure of the network's architecture and the low-level gradients of the input image itself. The unique nuances of the higher-level weights and the specific target class are mathematically squashed out. This explains why standard input-dependent saliency methods often exhibit a weak dependence on actual model parameters, capturing image silhouettes rather than internal decision logic, as argued by Akhtar et al. in Rethinking Interpretation: Input-Agnostic Saliency Mapping of Deep Visual Classifiers.

Alternative Benchmarks: Remove and Retrain

If visual clarity is a misleading indicator of fidelity, how can we objectively measure whether an attribution method identifies features that the model actually relies upon?

To bypass the subjectivity of visual inspection, Hooker et al. proposed the ROAR (RemOve And Retrain) framework in A Benchmark for Interpretability Methods in Deep Neural Networks.

1. Generate Saliency Map ---> 2. Remove Top "Important" Pixels ---> 3. Retrain Model From Scratch
                                                                            |
                                                                  Does accuracy drop?

Instead of simply masking pixels on a trained model (which alters the data distribution and introduces artifacts), ROAR removes the top fraction of "important" pixels identified by a saliency method across the entire dataset, and then retrains the model from scratch.

The intuition is elegant: if a saliency method truly identifies features essential to the model's predictive capacity, removing those features before training must degrade the retrained model's performance. When evaluated via ROAR, many popular gradient-based saliency methods performed no better than a completely random selection of pixels, confirming that they failed to isolate the true feature dependencies of the network.

Designing Sanity-Preserving Explanations

The realization that popular explanations fail basic debugging tests (Adebayo et al.) has forced the interpretability community to design new, sanity-preserving attribution methods.

Rather than relying on modified backpropagation through individual pixels, newer techniques enforce parameter sensitivity by design:

  • Region-Based and Multi-Scale Fusion: Methods like XRAI (Kapishnikov et al.) combine multi-scale segmentation with Integrated Gradients to shift focus from individual pixels to image regions. Similarly, CAMERAS (Jalwana et al.) fuses activation maps and gradients across multiple input scales, generating high-resolution maps that successfully pass parameter randomization tests.
  • Information-Theoretic Inversion: In Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution, Deng et al. proposed layer-by-layer network inversion designed to maximize mutual information. This ensures that the generated heatmaps remain mathematically bound to the specific learned parameters and target classes.

Concurrently, the evaluation tools themselves are becoming more robust. In A Fresh Look at Sanity Checks for Saliency Maps, Hedström et al. introduced Smooth MPRT (sMPRT) to denoise evaluations through sampling, and Efficient MPRT (eMPRT), which replaces brittle similarity metrics with measures of explanation complexity. These advancements ensure that as deep learning models are deployed in high-stakes environments, our tools for debugging them are truly explaining the model's mind, not just the image's edges.

Key Foundational Work

Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018) — contributed 4 concept elements to this direction.
Sanity Checks for Saliency Maps (2018) — contributed 2 concept elements to this direction.
XRAI: Better Attributions Through Regions (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_sanity_checks_1b3c6014,
  title = {explanation sanity checks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-sanity-checks/rdr_1b3c6014bfb942f4bd1b1f48f95d6fc9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_explanation_sanity_checks_1b3c6014_v1,
  title = {explanation sanity checks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-sanity-checks/rdr_1b3c6014bfb942f4bd1b1f48f95d6fc9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake