When we deploy post-hoc explainability methods like SHAP, LIME, or Integrated Gradients to understand why a deep learning model made a specific prediction, we expect a coherent answer. However, a fundamental crisis has emerged in explainable AI (XAI): different explanation methods frequently yield wildly contradictory feature importance rankings for the exact same model and input. This phenomenon, formalized as the "disagreement problem" by Satyapriya Krishna et al. in The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective, forces practitioners to rely on unscientific intuition to decide which explanation to trust.
Furthermore, this fragility is not just a property of the explanation algorithms themselves. As Marc-Etienne Brunet et al. demonstrate in Implications of Model Indeterminacy for Explanations of Automated Decisions, two models trained on the same data with identical test performance can produce completely different explanations due to arbitrary training choices—a challenge known as "explanatory multiplicity." Similarly, Dylan Slack et al. show in On the Lack of Robust Interpretability of Neural Text Classifiers that models differing only by their random initial seed can yield entirely divergent feature attributions.
To resolve this ambiguity, a new research direction focuses on explanation consistency metrics and multi-method consensus. By mathematically quantifying where explainers agree and disagree, researchers can filter out algorithmic noise and isolate the true underlying logic of black-box models.
Mathematically Quantifying the Rift
To move beyond qualitative "gut checks," researchers have developed a suite of metrics to rigorously measure how much distinct interpretability methods agree or diverge.
- Top-$k$ Feature Overlap and Jaccard Similarity: Instead of looking at the entire feature spectrum, metrics often isolate the top-$k$ most influential features. Dylan Slack et al. utilize Jaccard similarity to measure the intersection over union of these top feature sets, revealing how consistently different algorithms highlight the same core variables.
- Rank Correlation and Distribution Distance: To capture the exact ordering of feature importance, benchmarking frameworks like the one proposed by Vinitra Swamy et al. in Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs use Spearman or Pearson rank correlation alongside distribution distance metrics to compare feature attributions and counterfactual generators.
- Spatial and Continuous Agreement: In spatial domains (such as medical imaging or neuroanatomy), researchers use metrics like the DICE score (Maxim Kan et al. in Interpretable Deep Learning for Pattern Recognition in Brain Differences Between Men and Women) or Jensen-Shannon divergence (Lars Nieradzik et al. in Challenging the Black Box: A Comprehensive Evaluation of Attribution Maps of CNN Applications in Agriculture and Forestry) to quantify the exact spatial alignment between different attribution maps.
Why Explainers Disagree
The root of the disagreement lies in the fact that different interpretability methods approximate a model's internal logic through entirely different mathematical lenses. For instance, Michael Neely et al. in A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing analyze the disconnect between raw attention weights and post-hoc feature attributions in Transformer models, showing that they rarely capture the same signal.
A perturbation-based method (which masks inputs to observe output shifts) and a gradient-based method (which computes backpropagated gradients) probe different facets of the model's decision boundary, as noted by Charles A. Ellis et al. in Novel methods for elucidating modality importance in multimodal electrophysiology classifiers. Because each method introduces its own algorithmic bias, a single explanation map is often more reflective of the explainer's math than the model's true internal logic.
Building Robustness Through Multi-Method Consensus
Rather than searching for a single "perfect" explainer, the field is shifting toward cross-method validation and ensemble techniques. The core intuition is simple: if multiple mathematically distinct explanation methods converge on the same features, those features are highly likely to reflect genuine model logic rather than algorithmic artifacts.
Voting Theory and Algorithmic Consensus
In Evaluating Neuron Interpretation Methods of NLP Models, Yimin Fan et al. apply voting theory to aggregate individual neuron interpretations. They demonstrate that despite disparate underlying mathematics, top methods achieve a 60% consensus, establishing a reliable pseudo-ground truth without relying on external human labels.
Dynamic Top-$k$ Smoothing
A major source of artificial disagreement is the arbitrary choice of a fixed $k$ (e.g., forcing an algorithm to show the "top 5 words"). Jonathan Kamp et al. address this in Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods by dynamically identifying "peaks" in attribution scores. This flexible thresholding significantly reduces the measured disagreement problem by aligning the evaluation with natural variations in feature importance.
Cross-Method Ensembles
In high-stakes domains like malware classification or industrial scheduling, researchers combine game-theoretic attribution (SHAP) with permutation importance or counterfactuals to cross-validate decisions (Quincy Card et al.; Alessandro Immordino et al.). Yongduan Song et al. note in CIS Publication Spotlight [Publication Spotlight] that generating a concordance of explanations significantly increases user and expert trust compared to relying on a single method.
Global Rule Synthesis
To resolve local contradictions across an entire dataset, Andrea Agiollo introduces the GELPE framework in From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers. GELPE distills local feature importance scores from multiple disagreeing explainers into a single, globally consistent logic program, transforming fragmented local insights into robust global rules.
Standardizing Evaluation Frameworks
To prevent fragmented methodologies, recent work focuses on centralized repositories and multi-model benchmarks. Nils Feldhus et al. introduced Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools, a massive open-source data hub providing standardized tools to calculate rank correlations across hundreds of thousands of pre-computed explanations.
Similarly, Mengnan Du et al. developed the $M^4$ benchmark in M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models, which evaluates explanations against an ensemble of different well-trained architectures to establish robust, cross-model feature agreement. By shifting the paradigm from individual explanation maps to multi-method consensus, the community is building a more rigorous, verifiable foundation for trust in machine learning.
Key Foundational Work
Interpretable Deep Learning for Pattern Recognition in Brain Differences Between Men and Women (2020) — contributed 1 concept elements to this direction.
On the Lack of Robust Interpretability of Neural Text Classifiers (2021) — contributed 1 concept elements to this direction.
Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools (2021) — contributed 1 concept elements to this direction.