When we deploy deep learning models in high-stakes domains like healthcare or autonomous driving, we cannot rely on accuracy alone. We need to know why a model made a specific decision. To do this, researchers have developed a wide array of explainability (XAI) methods—such as saliency maps, class activation mappings, and feature attributions—designed to highlight the input features most critical to a model's output.
To determine which explanation method is the most truthful, or "faithful," to the underlying model, the machine learning community relies on quantitative evaluation metrics. However, this evaluation ecosystem faces a fundamental crisis: different metrics frequently produce completely contradictory rankings of the same explanation methods. An attribution method crowned as state-of-the-art by one metric might be dismissed as no better than random noise by another.
This rank inconsistency prevents researchers from establishing a clear hierarchy of reliable interpretability techniques. To understand why this disagreement happens, we must look beneath the surface at the conflicting mathematical properties these metrics reward, the out-of-distribution traps they fall into, and the software variances that compound the confusion.
The Ground Truth Paradox
The core difficulty of evaluating model explanations is the lack of an objective ground truth. We do not actually know what internal logic a black-box neural network used; if we did, we would not need the explanation method in the first place.
As a result, researchers evaluate the evaluators by proposing proxy metrics. As shown by Chun Sik Chan et al. in A Comparative Study of Faithfulness Metrics for Model Interpretability Methods and Anna Hedström et al. in The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus, when these proxy metrics are pitted against one another, they routinely disagree.
This issue is not isolated to computer vision or natural language processing; it is a systemic property of validation metrics across machine learning. For instance, Martijn Gösgens et al. demonstrate in Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures that standard clustering validation measures disagree on which clustering is superior in roughly 40% of practical cases. Similarly, Zihan Zhu et al. find in Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators that traditional performance estimators yield deeply conflicting conclusions in low-variance scenarios.
In the context of XAI, this lack of consensus means that the perceived quality of an explanation is often an artifact of the metric chosen rather than the inherent truthfulness of the explanation itself, as noted by Jia Li et al. in A Data-Driven Metric for Comprehensive Evaluation of Saliency Models and Lars Nieradzik et al. in Challenging the Black Box: A Comprehensive Evaluation of Attribution Maps of CNN Applications in Agriculture and Forestry.
Incompatible Mathematical Lenses
Why do these metrics disagree so fundamentally? The first reason is that different metrics evaluate entirely distinct mathematical properties of an output map under the generic label of "quality."
Consider saliency models evaluated on human eye-fixation data. As Matthias Kuemmerer et al. explain in Consistent Saliency Benchmarking: How One Model Can Win on All Metrics, different metrics reward different mathematical structures. Some metrics reward continuous probability densities, while others reward discrete, sparse fixation points. The authors show that by treating saliency explicitly as a probability density and using Bayesian utility theory to modify the output map for each specific metric, a single underlying model can actually win on all benchmarks. Without this explicit mathematical translation, the model rankings remain highly inconsistent.
In feature attribution, metrics often pull explanations in opposite directions. Johannes Maeß et al. use spectral analysis in Uncovering the Structure of Explanation Quality with Spectral Analysis to reveal a fundamental trade-off between two competing properties:
- Stability: How robust the explanation is to minor input noise.
- Target Sensitivity: How precisely the explanation changes when the target class changes.
An explanation map that optimizes for stability (like smoothing out attributions) will naturally perform poorly on metrics that reward target sensitivity, and vice versa. Because a single explanation map cannot maximize both simultaneously, as discussed by Oren Barkan et al. in Improving LLM Attributions with Randomized Path-Integration, the choice of metric inherently biases the evaluation toward one property over the other.
Furthermore, Marvin N. Wright et al. demonstrate in Towards Understanding the Disagreement Problem in Neural Network Feature Attribution that rank-based metrics are highly unstable because different attribution algorithms shift the baseline "zero point" of feature relevance in different ways. When a model's internal processing shifts due to debiasing or architectural changes, establishing a consistent baseline for comparing these scores becomes incredibly difficult, a phenomenon also documented by Yifan Wang et al. in Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?.
The Perturbation Trap and Out of Distribution Artifacts
The most popular empirical metrics evaluate an explanation through feature perturbation. The intuition is simple: if an explanation method claims that a specific pixel or word is highly important to the model's prediction, then removing that feature should cause the model's confidence to plummet (a property often called Comprehensiveness or Deletion score). Conversely, keeping only that feature should be enough to maintain the prediction (Sufficiency or Insertion score).
This intuitive mechanism, however, contains a fatal flaw: feature erasure creates Out-of-Distribution (OOD) data.
[Original Image] ---> Model predicts "Cat" (Confidence: 95%)
|
v (Explanation identifies ears as important)
[Ears Blacked Out] -> Model confidence drops to 10%
Did the confidence drop because the ears were truly the core feature, or because a harsh black square created an unnatural artifact that broke the neural network's internal representations?
As Fan Yin et al. argue in On the Sensitivity and Stability of Model Interpretations in NLP and Xu Zheng et al. show in F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI, standard fidelity metrics frequently fail because they cannot distinguish between true feature importance and OOD shock. This lack of monotonic sensitivity can lead metrics to assign higher faithfulness scores to completely degraded or random explanations than to genuine ones.
This OOD vulnerability makes perturbation metrics highly sensitive to the choice of the "neutral" baseline used to mask features. Lukas Brunke et al. demonstrate this fragility in Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison. By evaluating the same explanation method using different masking baselines—such as replacing pixels with black constants, uniform noise, or localized blurs—the authors found that the performance rankings of different explanation methods completely reversed.
Because these metrics are fundamentally biased toward erasure-based attribution frameworks, as noted by Yiming Ju et al. in A Hierarchical Explanation Generation Method Based on Feature Interaction Detection, a method's perceived superiority is often an artifact of the evaluation setup rather than its actual explanatory power.
Engineering and Software Variance
The disagreement problem is further compounded by unnormalized metrics and software engineering realities.
Metrics like the Area Over the Perturbation Curve (AOPC) measure raw confidence drops. However, Joakim Edin et al. point out in Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attributions Explainability that the theoretical minimum and maximum bounds of these curves vary drastically depending on the specific model architecture and input sample. When the authors introduced a normalized version (NAOPC) that scales scores based on these model-specific bounds, the cross-model interpretability rankings completely reversed, suggesting that many prior comparative studies may have reached misleading conclusions.
At the implementation level, the landscape is equally fragmented. Phuong Le et al. conducted a systematic survey in Benchmarking eXplainable AI - A Survey on Available Toolkits and Open Challenges, evaluating 17 XAI toolkits. They discovered that the exact same metric can produce wildly different numerical results—sometimes differing by multiple orders of magnitude—across different software libraries. This variance is driven by subtle differences in default hyperparameters, such as perturbation step sizes or masking thresholds, making direct cross-paper comparisons highly unreliable.
Pathways to Resolution
To navigate this maze of conflicting signals, researchers are shifting away from a reliance on single, unnormalized metrics toward more robust evaluation paradigms.
Aggregate Ranking Schemes
Recognizing that individual metrics capture isolated dimensions of explanation quality, benchmarks like LATEC, introduced by Lukas Klein et al. in Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics, evaluate methods across dozens of metrics simultaneously. By employing an "aggregate-then-rank" scheme, they mitigate single-metric bias and identify robust, versatile explanation methods (such as Expected Gradients) that consistently perform well across the board.
In-Distribution Perturbations
To eliminate the OOD artifacts that break perturbation metrics, Lior Wolf introduces a generative approach in A Meaningful Perturbation Metric for Evaluating Explainability Methods. Instead of deleting pixels or filling them with gray constants, the framework uses generative models (like Stable Diffusion) to realistically "inpaint" the masked regions. This ensures the modified image remains natural and within the model's training distribution, allowing changes in confidence to reflect true feature importance rather than model confusion.
Ground-Truth Injection
Rather than trying to guess what a standard model is looking at, Mengxi Ya et al. propose a method to force the model's internal logic into a known state. In Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark, the authors inject a backdoor watermark during training that forces the model to rely strictly on a precise, localized trigger feature. This creates a definitive, synthetic ground truth, allowing researchers to evaluate exactly how accurately an attribution method captures that known internal state without relying on fragile proxy metrics.
Key Foundational Work
Consistent Saliency Benchmarking: How One Model Can Win on All Metrics (2018) — contributed 2 concept elements to this direction.
A Data-Driven Metric for Comprehensive Evaluation of Saliency Models (2015) — contributed 1 concept elements to this direction.