When a deep learning model identifies a bird in an image or diagnoses a disease from a transcript, we often ask "Why?" Explainability (XAI) methods respond with heatmaps or highlighted words, suggesting these features drove the decision. But there is a trap: an explanation can be plausible (it makes sense to a human) without being faithful (it accurately reflects the model's internal logic).
To bridge this gap, researchers have converged on a simple, powerful intuition: if a feature is truly responsible for a prediction, the prediction should vanish when that feature is removed. This process, known as feature ablation or perturbation, has become the gold standard for auditing the "black box."
The Causal Test for Faithfulness
The fundamental problem in XAI is the lack of "ground truth." As Fan Yang and Xia Hu argue in Evaluating Explanation Without Ground Truth in Interpretable Machine Learning, we cannot simply look at an explanation and decide if it is correct. Instead, we must treat the explanation as a hypothesis and test it through intervention.
The logic follows a "deletion" protocol:
- Rank features (pixels, words, or audio snippets) by their supposed importance.
- Systematically remove the top-ranked features.
- Observe the degradation in model performance.
If the model’s confidence drops sharply after removing only a few "important" features, the explanation is faithful. If the model remains unfazed, the explanation is likely a hallucination—a visual artifact that looks convincing to humans but ignores the model's actual reasoning. Gabriëlle Ras and colleagues emphasize in their Field Guide for the Uninitiated that this retraining or re-testing is the only way to verify if the model's logic is truly captured.
Quantifying the Decline
To move beyond qualitative "eye-balling" of heatmaps, researchers use metrics that turn these interventions into curves. Hillary Ngai and Frank Rudzicz introduced the Feature Attribution Dropping (FAD) curve in Doctor XAvIer. By plotting accuracy against the percentage of features removed, they create a signature of faithfulness: the steeper the curve, the more accurately the XAI method has identified the model's "load-bearing" features.
Standardization is the next frontier. Toolkits like Quantus, developed by Anna Karin Hedström and Wojciech Samek, provide dozens of these metrics—such as sufficiency and comprehensiveness—to ensure that XAI evaluation is a rigorous science rather than a "guessing game" (Quantus: An Explainable AI Toolkit).
Faithfulness Across Modalities
This ablation logic is remarkably versatile, appearing across diverse domains of AI:
- Natural Language Processing: Pepa Atanasova and Isabelle Augenstein found that gradient-based methods often prove more faithful than perturbation-based ones when tokens are systematically removed in text classification tasks (A Diagnostic Study of Explainability Techniques).
- Computer Vision: In the study of CLIP models, Maximilian Dreyer and Sebastian Lapuschkin used ablation to reveal that the model often relies on "shortcuts" like background colors rather than the actual objects described in text (From What to How).
- Audio: The Listen to Interpret (L2I) framework by Jayneel Parekh allows users to "hear" explanations by removing audio components. If removing a "dog bark" component kills the "dog" prediction, the explanation is verified (Listen to Interpret).
- Robotics: In PhysNav-DG, Trisanth Srinivasan uses visual masking to check if a robot's "Chain-of-Thought" explanation is actually grounded in its sensors. If the robot claims it sees a door but its confidence doesn't change when the door is masked, the system knows to stop trusting the AI.
The Plausibility Trade-off
A common fear in the field is that a perfectly faithful explanation might be unintelligible to humans because neural networks don't "think" like we do. However, Xiaolei Lu and Jianghong Ma suggest this may be a false dichotomy. Their empirical study, Does Faithfulness Conflict with Plausibility?, shows that certain methods can achieve high scores in both dimensions, suggesting we can have explanations that are both honest to the model and useful to the user.