Benchmarking Faithfulness in Feature Attribution Methods

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,313 papers1,696 conceptsfrozen 2026-10-04

When a deep neural network identifies a tumor in an MRI or flags a transaction as fraudulent, we naturally ask: Why? To answer this, researchers have developed a suite of "Explainable AI" (XAI) tools—like Layer-Wise Relevance Propagation (LRP), SHAP, and Integrated Gradients—that produce heatmaps or "saliency maps" highlighting the most important input features.

However, a beautiful heatmap can be a dangerous mirage. As Alexander Binder and Wojciech Samek pointed out in On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation, interpretability is often a subjective, human-centric property. The central challenge of the field is not just generating explanations, but proving they are faithful—that they accurately reflect the model's internal logic rather than just providing a "plausible-sounding" justification that satisfies a human observer.

The Ground Truth Gap

In standard machine learning, we evaluate a model by comparing its prediction to a ground-truth label. In explainability, we rarely have a "label" for the explanation itself. We know what the model predicted, but we don't know for certain which features it actually used to get there.

This lack of objective ground truth, highlighted by Eric Wallace and Matthew Gardner 0001 in Interpreting Predictions of NLP Models, creates a circularity problem: we use XAI to understand the model, but we need to understand the model to know if the XAI is working. Without a reference point, an explanation might be "plausible" (it looks like a cat's ears) but not "faithful" (the model actually focused on the background texture).

Testing Faithfulness via Perturbation

To break this circularity, researchers use perturbation analysis. The intuition is simple: if a feature is truly important to the model's decision, removing it should cause the model's confidence to drop.

In Evaluating the visualization of what a Deep Neural Network has learned, Wojciech Samek and colleagues formalized this by systematically "flipping" or blurring pixels identified as important. If the model's performance degrades rapidly when these pixels are removed, the explanation is considered faithful. Using this metric, they demonstrated that Layer-Wise Relevance Propagation (LRP) often provides more accurate insights than simple gradient-based methods.

Similarly, in the realm of Question Answering, Alona Sydorova and Benjamin Roth 0001 introduced a "fake facts" paradigm in Interpretable Question Answering on Knowledge Bases and Text. By injecting controlled, synthetic information, they could measure whether an XAI method correctly identified the specific (and known) source of the model's answer.

Unit Tests and Synthetic Shortcuts

Because real-world data is messy, researchers have turned to "unit tests" using synthetic datasets where the ground truth is mathematically guaranteed.

In Do Feature Attribution Methods Correctly Attribute Features?, Yilun Zhou and Serena Booth created datasets where only specific, known features were predictive. They found that many popular attribution methods failed to identify these features, even when the model was 100% accurate. This suggests that many XAI tools may be picking up on artifacts of the explanation algorithm itself rather than the model's reasoning.

Polina Zablotskaia and Jasmijn Bastings took a similar approach in text classification, using "shortcuts"—deliberate, simple patterns that a model could exploit. Their protocol, Will You Find These Shortcuts?, revealed that the "best" explanation method is highly dependent on the model architecture, warning against a one-size-fits-all approach to interpretability.

Sanity Checks and Robustness

If an explanation is faithful to a model, it should change when the model changes. Ming Fan and Yang Liu applied this logic to security in Can We Trust Your Explanations?. They proposed "sanity checks" for malware analysis, measuring:

  1. Stability: Does the explanation stay the same for similar inputs?
  2. Robustness: Does the explanation change if the model's weights are randomized?
  3. Effectiveness: Does the explanation actually help a human detect a threat?

Their findings echoed a growing consensus: black-box methods like LIME and SHAP can be fragile and inconsistent, while white-box methods like Integrated Gradients and LRP tend to be more reliable in high-stakes environments like cybersecurity (Alexander Warnecke et al., Evaluating Explanation Methods for Deep Learning in Security).

Aligning with Domain Knowledge

In specialized fields like medicine, "ground truth" can sometimes be borrowed from decades of scientific research. Johannes Rieke and Kerstin Ritter explored this in Visualizing Convolutional Networks for MRI-Based Diagnosis of Alzheimer's Disease. By comparing neural network heatmaps to known clinical patterns of brain atrophy, they could quantify whether the model was "looking" at the same regions a radiologist would.

Recent work has extended this by comparing AI heatmaps to massive meta-analyses of neuroimaging studies, providing a quantitative benchmark for fidelity (Deep neural network heatmaps capture Alzheimer's disease patterns).

The Path Forward

The research direction is clear: the field is moving away from "looking at heatmaps" toward a rigorous, quantitative science of validation. Whether through causal reasoning frameworks like $CF^2$ (Juntao Tan et al., Learning and Evaluating Graph Neural Network Explanations) or diagnostic properties for NLP (Pepa Atanasova et al., A Diagnostic Study of Explainability Techniques for Text Classification), the goal is to ensure that when a model "shows its work," we can actually trust what we see.

Papers in this version

Showing 60 of 1,313 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 976 papers
  2. Version 1 · 2026-10-04 · 1313 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_quality_metrics_33fa5b87,
  title = {explanation quality metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-quality-metrics/rdr_33fa5b87060c46e59746885c119f7559},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_benchmarking_faithfulness_in_feature_att_33fa5b87_v1,
  title = {Benchmarking Faithfulness in Feature Attribution Methods},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/benchmarking-faithfulness-in-feature-attribution-methods/rdr_33fa5b87060c46e59746885c119f7559/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}