Parameter Randomization Tests for Model Explanations

42 concepts35 papers2 proposals

When we look at a feature attribution map—a heatmap highlighting the pixels or words that supposedly caused a deep learning model to make a prediction—our brains are highly adept at finding patterns. If a model classifies an image as a "Siberian Husky" and the heatmap highlights the dog's fluffy ears and sharp eyes, we instinctively conclude that the explanation is faithful: the model has learned to look at the correct biological features.

However, this visual coherence can be a dangerous illusion.

Foundational work by Adebayo et al. exposed a troubling reality: several popular attribution methods produce nearly identical heatmaps whether the model is a highly trained expert or a collection of completely randomized weights. Instead of explaining the model's internal logic, these methods act merely as sophisticated edge detectors, responding entirely to the structural priors of the input data rather than the learned parameters of the network.

To combat this, researchers have established a rigorous paradigm of parameter-sensitive attribution testing. By systematically perturbing or randomizing model weights, we can verify whether an explanation is truly a window into a model's learned logic or just an expensive input filter.

The Core Mechanism: Cascading Randomization

To understand why an explanation must be sensitive to parameters, consider what happens when a neural network's weights are destroyed.

If you randomize the weights of the very top layer (the output classification layer), the model's capacity to predict a specific class vanishes. Its outputs become uniform noise. Logically, if an explanation method claims to show why the model predicted a specific class, the explanation must change drastically once that class-specific logic is erased.

Adebayo et al. formalized this intuition through cascading parameter randomization. The procedure operates like a controlled demolition of the network's learned hierarchy:

  1. Isolate the baseline: Generate an attribution map for a trained model on a given input.
  2. Layer-by-layer destruction: Randomize the weights of the network sequentially, starting from the output layer and moving backward toward the input layer.
  3. Measure degradation: At each step, generate a new attribution map and compute its structural similarity to the baseline map.
[Trained Model] ───> Generates Baseline Saliency Map
       │
       ▼ (Randomize Output Layer)
[Model State 1] ───> Does the map change? (If NO, method fails class sensitivity)
       │
       ▼ (Randomize Hidden Layers Progressively)
[Model State 2] ───> Does the map degrade? (If NO, method fails parameter sensitivity)
       │
       ▼
[Fully Random]  ───> Map should look like random noise, not clean input edges

If the explanation map remains unchanged as the weights are scrambled, the method fails the sanity check. It proves that the visualization is independent of the model's internal state Adebayo et al.. As a systematic review by Nauta et al. notes, this form of parameter randomization has become a cornerstone metric for quantifying explanation correctness and faithfulness.

Why Do Explanations Lie? The Rank-1 Convergence Trap

To design better interpretability tools, we must understand why traditional methods fail these sanity checks. The culprit usually lies in how gradients are manipulated during backward propagation.

Methods like Guided Backpropagation and Layer-wise Relevance Propagation (LRP) modify standard backpropagation to prevent negative gradients or negative relevance from flowing backward, aiming to produce cleaner, less noisy visualizations. However, Sixt et al. provided a mathematical proof showing that these modifications force the backward pass to act as a chain of non-negative matrices.

As the backward flow passes through dozens of deep layers, these non-negative matrices mathematically converge to a rank-1 state. Once the gradient reaches this state, the influence of the higher-layer weights and the target class label is completely washed out. The backward pass effectively "forgets" what class it was supposed to explain and instead outputs a function determined almost entirely by the lower-layer weights and the input image gradients. This is why they look like beautiful edge detectors: they are mathematically constrained to highlight input boundaries, regardless of whether the model is trained or random.

Designing Parameter-Sensitive Attributions

Armed with parameter randomization as a diagnostic tool, researchers have developed new attribution frameworks designed from the ground up to be faithful to model weights.

1. Perturbation and Masking

Instead of relying on modified backward passes, perturbation-based methods treat the model as an active function. By applying smooth masks or finding "extremal perturbations" that maximize or minimize the model's output, these methods directly probe how changes in the input affect the specific learned parameters Fong et al.. Similarly, mask-based explanation models like the LXR framework developed by Barkan et al. demonstrate high sensitivity to internal weights because they optimize masks against the live, un-randomized outputs of the underlying architecture.

2. Information-Theoretic and Game-Theoretic Probes

Another path to parameter sensitivity is anchoring explanations to predictive information or cooperative game theory. For instance, the InputIBA method introduced by Zhang et al. applies an information bottleneck directly to the input domain to isolate features carrying the most predictive information. Because it targets the model's actual predictive capacity, its explanations degrade immediately if the model parameters are randomized.

From a game-theoretic perspective, the SUNY framework proposed by Lin et al. uses Shapley values to evaluate features based on necessity and sufficiency. By intervening on features and observing the delta in model activations, it ensures the resulting heatmaps reflect the internal logic of the trained network rather than simple input gradients.

3. Generative and Adversarial Baselines

Standard gradient methods often suffer because they evaluate gradients at a single, static point. Methods like Adversarial Gradient Integration (AGI), developed by Pan et al., bypass this by integrating gradients along a path from an adversarial starting point back to the target input. Because adversarial paths are explicitly defined by the model's unique decision boundaries, AGI naturally passes parameter randomization tests. In a similar vein, Shih et al. utilize generative adversarial networks (GANMEX) to construct class-specific baselines, ensuring that the resulting one-vs-one attributions are fundamentally dependent on the classifier's learned parameters.

Nuances in Testing: Bottom-Up vs. Top-Down

As the field has matured, the sanity checks themselves have faced scrutiny. Binder et al. argued that traditional top-down randomization can sometimes unfairly penalize valid explanation methods due to structural properties of certain network architectures.

To make these evaluations more robust, Hedström et al. revisited the Model Parameter Randomisation Test (MPRT) and demonstrated that a bottom-up disruption (randomizing layers starting from the input moving toward the output) is often more effective. This bottom-up approach prevents low-level features from early layers from persisting as residual architectural biases in the explanation. They also introduced Smooth MPRT to denoise evaluations, providing a cleaner metric for distinguishing true feature detection from structural artifacts.

Top-Down Randomization (Traditional):
[Input] ──> [Layer 1] ──> [Layer 2 (Scrambled)] ──> [Output (Scrambled)]
*Early structural biases can still leak into the backward pass*

Bottom-Up Randomization (Alternative):
[Input] ──> [Layer 1 (Scrambled)] ──> [Layer 2] ──> [Output]
*Disrupts the representation immediately, preventing structural leakage*

Universality Across Domains

While parameter sensitivity testing originated in computer vision, its core philosophy has spread across deep learning applications wherever the stakes of explanation failure are high:

  • Medical Imaging: In high-stakes clinical tasks, such as prostate tumor segmentation, Gunashekar applies cascaded randomization tests to ensure that Grad-CAM heatmaps are functionally linked to trained parameters rather than image artifacts. Kuijf et al. emphasize parameter sensitivity verification as a mandatory sanity check to prevent clinicians from trusting unfaithful edge-detecting explanations.
  • Scientific Discovery: When deep networks are used to extract cosmological data from weak lensing maps, Zorrilla Matilla et al. use parameter randomization to verify whether saliency maps are capturing genuine physical logic (like cosmic voids or peaks) or merely acting as generic image filters.
  • Alternative Architectures and Paradigms: Nguyen extended weight integration to Spiking Neural Networks (SNNs) to ensure that temporal spike attributions reflect learned synaptic weights rather than input frequency patterns. For similarity-based explanations (which explain a model by showing similar training examples), Hanawa et al. deployed model randomization to confirm that the selected examples reflect internal representations rather than raw dataset statistics.
  • Large Language Models: The challenge persists in modern architectures. Heap demonstrated that automated interpretability metrics in mechanistic interpretability often fail to distinguish between features extracted from fully trained Transformers and those from completely random null models, underscoring that parameter-sensitivity testing remains an active, critical frontier.

Key Foundational Work

Sanity Checks for Saliency Maps (2018) — contributed 3 concept elements to this direction.
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018) — contributed 1 concept elements to this direction.
Learning Performance of a Machine Translation System: a Statistical and Computational Analysis (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_sensitive_attribution_c010f4b8,
  title = {parameter-sensitive attribution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sensitive-attribution/rdr_c010f4b851cc47209e7bca2bdb281392},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_sensitive_attribution_c010f4b8_v1,
  title = {parameter-sensitive attribution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sensitive-attribution/rdr_c010f4b851cc47209e7bca2bdb281392/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake