Deep neural networks are notoriously opaque. To understand their decisions, researchers have developed feature attribution methods—such as saliency maps, integrated gradients, and attention weights—that highlight which parts of an input (pixels, tokens, or graph nodes) were most important to a model's prediction. However, these post-hoc explanations face a fundamental crisis: how do we know if an explanation is faithful to the model's internal logic, or if it is simply generating a human-plausible story that the model did not actually use?
To bridge this gap, the field has shifted toward quantitative validation frameworks. The core intuition is simple: if an attribution method accurately identifies the features driving a model's prediction, then systematically manipulating those features should predictably alter the model's behavior. By treating explanation validation as a causal intervention problem, researchers can objectively measure how closely an interpretation aligns with a model's actual decision-making process.
The Mechanics of Perturbation: Deletion and Insertion
The most direct way to validate an explanation is to alter the input based on the explanation's guidance. This is typically operationalized via two complementary tests: deletion (testing for necessity) and insertion (testing for sufficiency).
In a deletion test, features are removed or masked in descending order of their attributed importance. If the explanation is faithful, the model's confidence should drop sharply because its primary levers of evidence are being stripped away. As formalized by Jay DeYoung et al. in ERASER: A Benchmark to Evaluate Rationalized NLP Models, comprehensiveness measures this exact drop in confidence when the core rationale is removed. Conversely, sufficiency evaluates whether the highlighted features alone contain enough information for the model to retain its original prediction when all other context is deleted.
Rather than evaluating a single static mask, researchers often plot the model's response across a continuous trajectory of feature removal. Dong Nguyen 0002 (Comparing Automatic and Human Evaluation of Local Explanations for Text Classification) demonstrates the utility of perturbation curve analysis, where the Area Over the Curve (AOC) during iterative deletion quantifies how efficiently an explanation isolates impactful components.
Model
Confidence
^
1.0 |==========\
| \ <- Unfaithful Explanation (Slow drop)
| \____________
|----\
| \ <- Faithful Explanation (Sharp drop)
0.0 +------\------------------->
0% 20% 40% 60%
Features Removed (Highest Importance First)
Similarly, Zhongang Qi et al. (Visualizing Deep Networks by Optimizing with Integrated Gradients) utilize both deletion and insertion curves to benchmark visual explanations, showing that high-quality attributions lead to rapid confidence drops during deletion and rapid gains during insertion.
Quantifying Magnitude and Sensitivity
While deletion curves evaluate whether the rank order of important features is correct, they do not necessarily verify if the magnitude of the attribution scores matches the feature's true causal weight. A faithful explanation should not only tell you what is important, but exactly how much it matters.
To formalize this relationship, Marco Ancona et al. (Towards better understanding of gradient-based attribution methods for Deep Neural Networks) introduced the Sensitivity-n metric. This approach measures the correlation between the sum of attribution scores for a random subset of $n$ features and the actual change in the model's output when those $n$ features are removed. If an attribution method possesses high fidelity, there will be a strong linear correlation between the predicted importance sum and the functional output shift across varying scales of feature occlusion.
Overcoming the Distribution Shift Pitfall
While perturbation metrics offer an elegant window into a model's inner workings, they suffer from a major confounding factor: out-of-distribution (OOD) shift. When you mask a token or black out a pixel, you create an artificial input that looks fundamentally different from the training data. The model's confidence might drop not because you removed a critical feature, but simply because the model is confused by the unnatural artifact or zero-valued mask.
To isolate true feature importance from the confounding effects of distribution shift, Andreas Madsen et al. (Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining) proposed Recursive ROAR (Remove and Retrain). Instead of evaluating the base model on corrupted inputs, the authors iteratively mask the most "important" features across the entire dataset and retrain the model from scratch.
If the attribution method is honest, removing those features should permanently degrade the dataset's predictive value, forcing the retrained model to perform poorly. By measuring performance degradation on a cleanly retrained model, ROAR bypasses the OOD artifact problem, providing a robust benchmark for evaluating explanation faithfulness across diverse tasks and architectures.