To understand why a deep learning model made a specific prediction, researchers often use "feature attribution" methods—tools like SHAP, LIME, or Integrated Gradients that produce heatmaps or saliency maps highlighting the most influential parts of an input. But how do we know if these explanations are actually "faithful" to the model's internal logic?
The standard way to test an explanation is the Deletion Game: if a saliency map claims a specific pixel or word is important, removing it should cause the model's confidence to plummet. Conversely, an Insertion Game suggests that starting with a blank canvas and adding only the "important" features should be enough for the model to make a correct prediction.
However, a growing body of research suggests these metrics are fundamentally flawed. The act of removing or masking features creates "Frankenstein" inputs that the model never encountered during training, leading to a phenomenon known as Out-of-Distribution (OOD) shift.
The Problem of Off-Manifold Samples
When we "delete" a feature by replacing it with a black pixel, a zero-vector, or a mean value, we are not just removing information; we are changing the data distribution. As Richard Tomsett and Supriyo Chakraborty argue in their meta-analysis of saliency metrics, these artificial inputs often fall "off-manifold"—meaning they do not resemble the natural data the model was optimized for.
Because the model has no experience with these malformed inputs, its behavior becomes unpredictable. A drop in prediction confidence might not mean the removed feature was "important"; it might simply mean the model is confused by the noise. Tom Begley demonstrates that standard Shapley values often evaluate models on these "nonsense" data points, which can lead to misleading or even adversarially exploitable explanations.
Domain-Specific Artifacts
The OOD problem manifests differently across data modalities, but the underlying mechanism remains the same:
- Natural Language Processing: In NLP, deleting "important" tokens often results in ungrammatical or nonsensical strings. Fan Yin and Cho-Jui Hsieh point out that traditional faithfulness metrics cannot distinguish between a model losing a critical feature and a model failing because the input is no longer valid English.
- Graph Neural Networks: Removing edges or nodes to test a GNN explainer creates structural "Frankensteins." Xu Zheng and Farhad Shirani show that these broken graphs trigger unreliable outputs, making it difficult to measure the true fidelity of the explanation.
- Time Series: For temporal data, Aya Abdelsalam Ismail notes that masking features can break the temporal dependencies the model relies on, leading to accuracy drops that reflect distribution shifts rather than feature importance.
Building Robust Evaluation Frameworks
To fix these metrics, researchers are moving toward "on-manifold" or "distribution-aware" evaluation strategies. The goal is to ensure that the modified inputs used for testing remain within the realm of what the model understands.
1. Generative Infilling and Counterfactuals
Instead of replacing pixels with black squares, some researchers use generative models to "inpaint" or fill in the gaps with realistic data. Yingqiang Ge and Shuchang Liu 0001 propose Counterfactual Evaluation (CE), which uses targeted, realistic changes to features to see if they flip the model's prediction, rather than simply deleting them.
2. Robust Fidelity and Stochastic Sampling
Rather than a "hard" removal of features, Xu Zheng and Wei Cheng introduce robust fidelity metrics for graphs that use stochastic edge sampling. This approach mitigates the impact of sudden structural changes, providing a smoother and more accurate measure of faithfulness.
3. Masked Fine-Tuning
Perhaps the most direct solution is to train the model to handle masked inputs from the start. Andreas Madsen introduces Faithfulness Measurable Models (FMMs), which use masked fine-tuning to ensure that when a feature is removed during evaluation, the model's reaction is a genuine reflection of that feature's absence, not a reaction to an unseen OOD artifact. Similarly, the F-Fidelity framework by Farhad Shirani and Xu Zheng uses robustness-building to prevent information leakage and OOD errors during the evaluation process.
The Path Forward
The consensus across these 80 papers is clear: an explanation is only as good as the metric used to validate it. If our metrics are sensitive to noise, artifacts, and distribution shifts, they will continue to rank "plausible" explanations (those that look good to humans) over "faithful" ones (those that actually reflect the model). By prioritizing distributional robustness, researchers like Cheng-Yu Hsieh and Thomas FEL are transforming XAI from a field of visual intuition into a rigorous science of model auditing.