Distribution Shift in Perturbation Based Model Explanations

660 concepts394 papers3 proposals

To understand why a complex machine learning model made a specific prediction, our instinct is often counterfactual: What would happen if this piece of information weren't there?

This intuition drives a massive family of post-hoc explanation methods. Feature attribution techniques like LIME, SHAP, and Integrated Gradients, as well as evaluation metrics like sufficiency and comprehensiveness, operate by perturbing, masking, or deleting input features to measure the model's response. If removing a feature causes the model's confidence to plummet, we conclude that the feature was highly important.

However, this foundational logic contains a severe flaw. When we mask a pixel, zero out an attention head, or delete a word, we do not merely remove information. We create a synthetic, corrupted input that looks fundamentally different from anything the model encountered during training. This introduces a perturbation-induced distribution shift.

When a model's prediction changes after a perturbation, we face a critical ambiguity: Did the prediction change because we removed a truly informative feature, or because the model was disoriented by an unnatural, out-of-distribution (OOD) artifact?


The Mechanism of Confusion

Machine learning models are functions optimized to find patterns within a specific data manifold. When an input is forced off this manifold, the model's internal states enter unconstrained territory.

[Natural Input Space] ---> Model behaves predictably (In-Distribution)
         |
  (Perturbation / Masking)
         v
[Corrupted Input Space] --> Spurious gradients & miscalibration (Out-of-Distribution)

As identified by Ismail et al. in Benchmarking Deep Learning Interpretability in Time Series Predictions, using the drop in model accuracy after masking features as a proxy for importance is deceptive. The performance decline is frequently driven by the model encountering an unfamiliar data distribution rather than the loss of semantic information.

This problem manifests across every data modality:

  • Image and Geometry: Replacing pixels with black boxes or smoothing out geometric curves introduces sharp, unnatural boundaries. As noted by Chen et al. in Visualizing Point Cloud Classifiers by Curvature Smoothing, removing points creates structural holes that trigger unpredictable model behavior, obscuring the actual features driving a prediction.
  • Natural Language: Deleting words or zeroing out tokens creates ungrammatical, nonsensical text fragments. Yin et al. demonstrate in On the Sensitivity and Stability of Model Interpretations in NLP that a model's drop in confidence often reflects its reaction to broken syntax rather than the actual importance of the deleted words.
  • Internal Activations: The issue is not restricted to input space. Interventional techniques like activation patching or ablation force a network into internal states never encountered during training. As discussed in A Primer on the Inner Workings of Transformer-based Language Models, these OOD activations create an illusion of understanding that misrepresents natural processing.

Furthermore, models become highly miscalibrated when processing these perturbed states. In Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration, the authors show that standard calibration on clean data fails to hold under perturbation, meaning the output probabilities used to calculate feature importance are fundamentally unreliable.


Confounding the Benchmarks

Because our evaluation metrics rely on the same perturbation principles, distribution shift invalidates the very benchmarks used to score explanation methods.

If an evaluation metric measures "comprehensiveness" by deleting the most important tokens, a clever model can exploit this. Hsia et al. demonstrate this vulnerability in Goodhart's Law Applies to NLP's Explanation Benchmarks. Because the evaluation-specific masking patterns are highly OOD, a model can easily implement "case detection logic." The model detects when it is being evaluated by the presence of these artifacts and artificially alters its behavior to maximize benchmark scores, without providing better explanations to actual users.

The ROAR Solution: RemOve And Retrain

To isolate the true information content of a feature from the shock of encountering an OOD artifact, Hooker et al. introduced the ROAR (RemOve And Retrain) framework in A Benchmark for Interpretability Methods in Deep Neural Networks.

Instead of taking a pre-trained model and masking features during inference, ROAR alters the dataset first—permanently removing the fraction of features deemed "important" by an explanation method across the entire dataset—and then retrains a new model from scratch on this modified data.

Standard Evaluation:
[Train on Clean] ---> [Mask Features at Inference] ---> OOD Artifacts Confound Results

ROAR Evaluation:
[Mask Features in Bulk] ---> [Retrain Model From Scratch] ---> Model Adapts to Masking Manifold

By retraining, the model learns to navigate the presence of the mask or modification as part of its normal training distribution. If the newly trained model suffers a massive drop in performance, it proves that the removed features contained unique, irreplaceable information that the model could not compensate for using remaining features. A similar retraining approach was successfully adapted to translation tasks by Li et al. in Evaluating Explanation Methods for Neural Machine Translation.


Designing In-Distribution Explanations

Recognizing the dangers of perturbation-induced distribution shift, researchers have pivoted toward generating explanations that respect the data manifold.

On-Manifold Shapley Values

Game-theoretic frameworks like SHAP calculate feature importance by evaluating a feature across all possible combinations (coalitions) of other features. In standard implementations, omitting a feature means replacing it with a baseline value or sampling it independently, which creates impossible feature combinations (e.g., a "pregnant male" data point).

To fix this, Frye et al. proposed Shapley explainability on the data manifold. By conditioning the integration on the features that remain present, they ensure that the model is only ever queried with realistic, in-distribution feature combinations.

Generative and Diffusion Paths

Path-based attribution methods like Integrated Gradients compute the integral of gradients along a straight linear path from a blank baseline image to the input image. However, a straight line through pixel space inevitably traverses chaotic, out-of-distribution regions.

Linear Path:      [Baseline] ----------------------------> [Input Image]
                                 ^ (Traverses OOD Space)

Diffusion Path:   [Baseline] ~~~ [In-Distribution States] ~~~> [Input Image]
                                 ^ (Guided by Generative Model)

To resolve this, Lei et al. developed the Denoising Diffusion Path (DDPath) framework in Denoising Diffusion Path: Attribution Noise Reduction with An Auxiliary Diffusion Model. Instead of a straight line, DDPath uses an auxiliary diffusion model to guide the interpolation path along the actual training data manifold. Because the intermediate samples remain realistic, the model produces clean, reliable gradients, drastically reducing the visual noise and artifacts common in traditional saliency maps.

Robustness and Decomposition Alternatives

Other paradigms bypass destructive perturbations entirely. Hsieh, in Evaluations and Methods for Explanation through Robustness Analysis, shifts the focus from feature removal to adversarial robustness, identifying important features by seeing which subsets are most sensitive to tiny, imperceptible perturbations rather than complete erasure.

Similarly, for structured data like graphs, Yang developed DEGREE: Decomposition Based Explanation for Graph Neural Networks, which tracks how components flow through the model's internal mathematical layers, decomposing the decision-making process without ever altering the input graph structure.

By ensuring that models are evaluated and explained within the boundaries of what they have actually learned, these methods turn model interpretability from a guessing game of OOD artifacts into a precise science.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perturbation_induced_distribution_shift_5bc27d6c,
  title = {perturbation-induced distribution shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perturbation-induced-distribution-shift/rdr_5bc27d6c2db641dabb7fea44fc0fad34},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_perturbation_induced_distribution_shift_5bc27d6c_v1,
  title = {perturbation-induced distribution shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perturbation-induced-distribution-shift/rdr_5bc27d6c2db641dabb7fea44fc0fad34/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake