Consensus and Consistency Metrics for Feature Attribution

34 concepts30 papers

When we deploy post-hoc explainability methods like SHAP, LIME, or Integrated Gradients to understand why a deep learning model made a specific prediction, we expect a coherent answer. However, a fundamental crisis has emerged in explainable AI (XAI): different explanation methods frequently yield wildly contradictory feature importance rankings for the exact same model and input. This phenomenon, formalized as the "disagreement problem" by Satyapriya Krishna et al. in The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective, forces practitioners to rely on unscientific intuition to decide which explanation to trust.

Furthermore, this fragility is not just a property of the explanation algorithms themselves. As Marc-Etienne Brunet et al. demonstrate in Implications of Model Indeterminacy for Explanations of Automated Decisions, two models trained on the same data with identical test performance can produce completely different explanations due to arbitrary training choices—a challenge known as "explanatory multiplicity." Similarly, Dylan Slack et al. show in On the Lack of Robust Interpretability of Neural Text Classifiers that models differing only by their random initial seed can yield entirely divergent feature attributions.

To resolve this ambiguity, a new research direction focuses on explanation consistency metrics and multi-method consensus. By mathematically quantifying where explainers agree and disagree, researchers can filter out algorithmic noise and isolate the true underlying logic of black-box models.

Mathematically Quantifying the Rift

To move beyond qualitative "gut checks," researchers have developed a suite of metrics to rigorously measure how much distinct interpretability methods agree or diverge.

Why Explainers Disagree

The root of the disagreement lies in the fact that different interpretability methods approximate a model's internal logic through entirely different mathematical lenses. For instance, Michael Neely et al. in A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing analyze the disconnect between raw attention weights and post-hoc feature attributions in Transformer models, showing that they rarely capture the same signal.

A perturbation-based method (which masks inputs to observe output shifts) and a gradient-based method (which computes backpropagated gradients) probe different facets of the model's decision boundary, as noted by Charles A. Ellis et al. in Novel methods for elucidating modality importance in multimodal electrophysiology classifiers. Because each method introduces its own algorithmic bias, a single explanation map is often more reflective of the explainer's math than the model's true internal logic.

Building Robustness Through Multi-Method Consensus

Rather than searching for a single "perfect" explainer, the field is shifting toward cross-method validation and ensemble techniques. The core intuition is simple: if multiple mathematically distinct explanation methods converge on the same features, those features are highly likely to reflect genuine model logic rather than algorithmic artifacts.

Voting Theory and Algorithmic Consensus

In Evaluating Neuron Interpretation Methods of NLP Models, Yimin Fan et al. apply voting theory to aggregate individual neuron interpretations. They demonstrate that despite disparate underlying mathematics, top methods achieve a 60% consensus, establishing a reliable pseudo-ground truth without relying on external human labels.

Dynamic Top-$k$ Smoothing

A major source of artificial disagreement is the arbitrary choice of a fixed $k$ (e.g., forcing an algorithm to show the "top 5 words"). Jonathan Kamp et al. address this in Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods by dynamically identifying "peaks" in attribution scores. This flexible thresholding significantly reduces the measured disagreement problem by aligning the evaluation with natural variations in feature importance.

Cross-Method Ensembles

In high-stakes domains like malware classification or industrial scheduling, researchers combine game-theoretic attribution (SHAP) with permutation importance or counterfactuals to cross-validate decisions (Quincy Card et al.; Alessandro Immordino et al.). Yongduan Song et al. note in CIS Publication Spotlight [Publication Spotlight] that generating a concordance of explanations significantly increases user and expert trust compared to relying on a single method.

Global Rule Synthesis

To resolve local contradictions across an entire dataset, Andrea Agiollo introduces the GELPE framework in From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers. GELPE distills local feature importance scores from multiple disagreeing explainers into a single, globally consistent logic program, transforming fragmented local insights into robust global rules.

Standardizing Evaluation Frameworks

To prevent fragmented methodologies, recent work focuses on centralized repositories and multi-model benchmarks. Nils Feldhus et al. introduced Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools, a massive open-source data hub providing standardized tools to calculate rank correlations across hundreds of thousands of pre-computed explanations.

Similarly, Mengnan Du et al. developed the $M^4$ benchmark in M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models, which evaluates explanations against an ensemble of different well-trained architectures to establish robust, cross-model feature agreement. By shifting the paradigm from individual explanation maps to multi-method consensus, the community is building a more rigorous, verifiable foundation for trust in machine learning.

Key Foundational Work

Interpretable Deep Learning for Pattern Recognition in Brain Differences Between Men and Women (2020) — contributed 1 concept elements to this direction.
On the Lack of Robust Interpretability of Neural Text Classifiers (2021) — contributed 1 concept elements to this direction.
Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_method_interpretability_a9b1e942,
  title = {multi-method interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-method-interpretability/rdr_a9b1e942a61249b58cbc622563446c5c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_method_interpretability_a9b1e942_v1,
  title = {multi-method interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-method-interpretability/rdr_a9b1e942a61249b58cbc622563446c5c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake