When a machine learning model identifies a stop sign or flags a fraudulent transaction, we often ask "why?" Explainable AI (XAI) methods respond with saliency maps or feature attributions—visualizations that highlight the input parts most responsible for the decision. However, a growing body of research suggests that these explanations are often "plausible" (they look right to humans) without being "faithful" (they don't actually reflect the model's internal logic).
To solve this, researchers have converged on a rigorous way to "audit" explanations: perturbation-based fidelity metrics. By systematically removing or isolating the features an explainer claims are important, we can measure exactly how much the model's confidence drops, turning subjective visual appeal into objective, quantitative evidence.
The Core Mechanism: Deletion and Insertion
The intuition behind fidelity metrics is simple: if an explanation is faithful, the features it highlights should be the primary drivers of the model's output. Antonio Longa and colleagues in Explaining the Explainers in Graph Neural Networks formalize this through two primary lenses:
- Comprehensiveness (Deletion): If we remove the features identified as "highly important," the model's confidence in its original prediction should plummet. If the confidence stays high, the explanation failed to capture the features the model actually relied on.
- Sufficiency (Insertion/Isolation): If we show the model only the features identified as important (and mask everything else), the model should still be able to make the same prediction.
Rojina Kashefi et al. describe this in Explainability of Vision Transformers as measuring the "Area Under the Curve" (AUC) during iterative pixel deletion. A steep drop in the deletion curve indicates that the explainer successfully pinpointed the model's "load-bearing" features.
Why Visuals Can Be Deceptive
The need for these metrics arises because humans are easily fooled by "edge-detectors" or "saliency-like" visualizations that look intuitive but are functionally disconnected from the model. Jose Oramas and Benjamin Vandersmissen found in On The Coherence of Quantitative Evaluation of Visual Explanation that an explanation's perceived quality often depends more on its visual properties (like blurriness or sparsity) than its actual faithfulness.
Similarly, Romain Xu-Darme and colleagues demonstrated in Sanity checks for patch visualisation that "self-explaining" models like ProtoPNet often produce visualizations that highlight background noise. Only through perturbation-based faithfulness tests could they prove that the model was actually focusing on the object, even when the default visualization suggested otherwise.
Testing the "Sanity" of the Metrics
If we use metrics to judge explainers, what judges the metrics? Richard Tomsett, Dan Harborne, and their team at Sanity Checks for Saliency Metrics applied psychometric reliability tests to the metrics themselves. They discovered that fidelity metrics can be highly sensitive to implementation details—such as whether you "delete" a pixel by turning it black, blurring it, or replacing it with mean noise. This "meta-analysis" warns that while perturbation is the right tool, the way we perturb the data can introduce its own biases.
Expanding to Language and Graphs
This framework is now being used to unmask the behavior of more complex architectures:
- Large Language Models: In Can Large Language Models Explain Themselves?, Shiyuan Huang and colleagues used comprehensiveness and sufficiency to show that LLM "self-explanations" are often just plausible stories. The models mimic human explanation styles rather than revealing their raw computational process.
- Vision-Language Models: Walid Bousselham and Hilde Kuehne introduced DEX-AR, which measures changes in perplexity (internal confidence) rather than just label changes to evaluate faithfulness in generative contexts.
- Graph Neural Networks: Zhiqiang Zhong and Davide Mottin showed in On the Robustness of Post-hoc GNN Explainers to Label Noise that while GNNs might be robust to noise, the explainers we use to interpret them are often fragile, a fact only detectable through fidelity-based evaluation.
Why This Matters for Fairness
Fidelity isn't just a technical requirement; it's a prerequisite for trust. Yu Wang and Tyler Derr argue in Fairness and Explainability that we cannot achieve "procedural fairness" if our explanations are unfaithful. If a model's stated reasoning (the explanation) does not align with its actual predictive behavior (the fidelity), we cannot truly know if the model is relying on biased or discriminatory features. By quantifying faithfulness, we move from asking if an explanation looks good to proving that it is true.