Evaluating Explainable AI through Iterative Feature Removal

This direction is merged into another direction: perturbation-based attribution validation. Its last version stays readable and citable here.

Version 1 of 118 papers20 conceptsfrozen 2026-10-04

Deep neural networks are often criticized as "black boxes" because their internal decision-making processes are opaque. To address this, researchers have developed various feature attribution methods—such as saliency maps or heatmaps—that highlight which parts of an input (like pixels in an image or words in a sentence) were most influential to a model's prediction.

However, a critical question remains: How do we know if these explanations are actually "faithful" to the model's logic? A heatmap might look intuitive to a human, but it could be a "pretty picture" that doesn't actually reflect the features the model used. To solve this, researchers have converged on a quantitative framework: perturbation-based validation.

The Intuition of Faithfulness

The core idea behind validating an explanation is simple: if an attribution method correctly identifies the most important features, then removing those features should cause the model’s performance to collapse.

Imagine a model that classifies an image as a "dog." An explanation method highlights the dog's ears as highly relevant. To test this, we "perturb" the image by masking the ears. If the model’s confidence in the "dog" classification drops significantly, the explanation was likely faithful. If the model remains just as confident, the explanation was misleading; the model was likely looking at something else entirely.

Wojciech Samek, Alexander Binder, and Sebastian Bach formalized this in Evaluating the visualization of what a Deep Neural Network has learned. They demonstrated that by systematically removing pixels in order of their "relevance" and measuring the rate of accuracy degradation, we can objectively rank explanation methods. A faster drop in accuracy indicates a more accurate explanation.

From Pixels to Tokens and Beyond

This perturbation logic is remarkably versatile and has been applied across diverse data modalities:

Layer-wise Relevance Propagation

Across many of these comparative studies, Layer-wise Relevance Propagation (LRP) has emerged as a particularly robust method. Unlike simple gradient-based methods—which only show how a small change in the input affects the output—LRP redistributes the actual prediction value backward through the network layers.

In Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models, Klaus-Robert Müller and colleagues argue that LRP provides more intuitive and "cleaner" explanations than Sensitivity Analysis. Perturbation tests consistently show that LRP-identified features are more functionally necessary for the model's decision than those identified by gradients.

The Fragility of Perturbation

While perturbation is the gold standard for evaluation, it is not without its own pitfalls. Lukas Brunke and Nikhil George pointed out in Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison that the way you remove a feature matters.

If you replace a "relevant" pixel with a black square, the model might get confused not because the feature is gone, but because a weird black square appeared in the image. This "distribution shift" can make an explanation look better or worse than it actually is. To mitigate this, researchers like Laura Rieger have suggested aggregating multiple explanation methods to stabilize the results and reduce noise, as explored in Aggregating explainability methods for neural networks stabilizes explanations.

Why It Matters

The ability to quantitatively verify explanations allows us to use AI in high-stakes environments with more confidence. For instance, Xenia Demetriou and Matthia Sabatelli used these metrics to prove that we can prune 90% of a model's size without losing the "faithfulness" of its explanations (A Study on the Faithfulness of Feature Attribution Explanations in Pruned Vision-Based Multi-Task Learning).

By treating explainability as a measurable engineering property rather than a subjective visual preference, this research direction moves us closer to AI systems that are not just accurate, but truly understandable.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_evaluating_explainable_ai_through_iterat_267ac217,
  title = {Evaluating Explainable AI through Iterative Feature Removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/evaluating-explainable-ai-through-iterative-feature-removal/rdr_267ac217da654b0abe60e33fbfb7b163},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_evaluating_explainable_ai_through_iterat_267ac217_v1,
  title = {Evaluating Explainable AI through Iterative Feature Removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/evaluating-explainable-ai-through-iterative-feature-removal/rdr_267ac217da654b0abe60e33fbfb7b163/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}