Assessing XAI: Unveiling Evaluation Metrics for Local Explanation, Taxonomies, Key Concepts, and Practical Applications

Md Abdul Kadir, Amir Mosavi, Daniel Sonntag

TL;DR

This paper provides a comprehensive systematic review of evaluation metrics for eXplainable AI (XAI), moving beyond qualitative "gut feelings" toward rigorous quantitative benchmarks. By analyzing over 70 key studies, the authors propose two new taxonomies—one based on evaluation methodologies and another on practical applications—to help researchers select the right metrics for validating model transparency.

What Problem Does This Paper Address?

As deep learning models become more accurate, they also become more opaque "black boxes." While the field of XAI has exploded with methods to explain these models (like LIME or SHAP), there is a glaring lack of consensus on how to measure if an explanation is actually good.

Currently, many researchers rely on qualitative assessments or human-subject studies, which are prone to confirmation bias and are difficult to scale. This paper addresses the "evaluation gap" by systematizing the scattered landscape of XAI metrics, providing clear definitions for concepts like fidelity, faithfulness, and stability, and offering a structured way to choose metrics based on the specific use case (e.g., healthcare vs. autonomous driving).

Methods

The authors conducted a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines.

  1. Initial Search: They started with 6,122 articles from databases like Scopus and Google Scholar.
  2. Screening: Through multi-stage filtering (keywords, abstract review, and full-text analysis), they narrowed the pool down to 70 core articles that specifically proposed or rigorously applied XAI evaluation metrics.
  3. Taxonomy Generation: Based on this corpus, they developed two distinct taxonomies:
  4. Methodology-based: Categorizes metrics by how they work (e.g., Input Perturbation, Retraining, Ground Truth Comparison).
  5. Application-based: Categorizes metrics by what they aim to achieve (e.g., Trustworthiness, Traceability, User Satisfaction).

Datasets

As a systematic review, this paper does not introduce a new training dataset. Instead, it catalogs the diverse data types used across the 70 reviewed studies to validate XAI metrics. These include:

  • Medical Imaging: COVID-19 chest X-rays, CT scans, and DNA analysis.
  • Industrial/IoT: Power quality disturbances and hard disk drive predictive maintenance.
  • Autonomous Systems: Self-driving car decision-making and drone navigation.
  • General Benchmarks: Standard ML datasets used for "Pixel Flipping" and "Region Perturbation" tests.

Key Insights and Ideas

  • The "Faithfulness" Challenge: The paper highlights that "faithfulness"—the degree to which an explanation accurately reflects the model's true reasoning—is the most critical but difficult metric to capture.
  • Sensitivity vs. Infidelity: The authors distinguish between Sensitivity (how much an explanation changes when the input changes slightly) and Infidelity (the distance between the explanation and the actual model prediction change after a perturbation).
  • The Retraining Trap: One common evaluation method involves removing "important" features and retraining the model to see if accuracy drops. The authors note this is computationally expensive and can create "out-of-distribution" artifacts that mislead the evaluation.
  • Psychology Matters: The paper argues that because humans are the end-consumers of XAI, metrics must eventually bridge the gap between numerical scores (like AOPC) and psychological constructs like trust and mental models.

Key Results

The paper provides a "cheat sheet" of mathematical definitions for the most robust quantitative metrics currently available:

  • Pixel Flipping & AOPC (Area Over Perturbation Curve): Measures how quickly a model's confidence drops when highly-ranked features are removed.
  • ROAD (RemOve And Debias): An improved version of feature removal that uses noisy linear imputations to prevent the model from seeing "impossible" data.
  • Sensitivity-N: A metric that computes the correlation between the sum of feature attributions and the variation in the model's output.
  • IROF (Iterative Removal of Features): Evaluates the stability of explanations by measuring the area under the curve as superpixels are removed.

The authors found that while XAI research is growing exponentially (as shown in their trend analysis), the subset of papers focusing on metrics is growing much more slowly, indicating a field that is "building faster than it can measure."

Limitations and Future Work

  • Lack of Standardization: The authors acknowledge that there is still no "one-size-fits-all" metric. A metric that works for image saliency maps might be useless for tabular data or Natural Language Processing (NLP).
  • Computational Cost: Many of the most "faithful" metrics (like those involving retraining or massive perturbations) are too slow for real-time applications.
  • Human-in-the-loop Gap: While the paper focuses on quantitative metrics, the authors admit that a high "faithfulness" score doesn't always mean a human can understand the explanation. Future work needs to better integrate quantitative metrics with user-centric qualitative studies.
  • Proposed "Universal" Metric: The conclusion calls for the development of a theoretically robust, generalized metric that can be applied across different architectures (CNNs, Transformers, etc.) to allow for true benchmarking.

Summary updated

Abstract

Within the past few years, the accuracy of deep learning and machine learning models has been improving significantly while less attention has been paid to their responsibility, explainability, and interpretability.eXplainable Artificial Intelligence (XAI) methods, guidelines, concepts, and strategies offer the possibility of models' evaluation for improving fidelity, faithfulness, and overall explainability.Due to the diversity of data and learning methodologies, there needs to be a clear definition for the validity, reliability, and evaluation metrics of explainability.This article reviews evaluation metrics used for XAI through the PRISMA systematic guideline for a comprehensive and systematic literature review.Based on the results, this study suggests two taxonomy for the evaluation metrics.One taxonomy is based on the applications, and one is based on the evaluation metrics.
BibTeX
@misc{kadir2023assessing,
  author = {Md Abdul Kadir and Amir Mosavi and Daniel Sonntag},
  title = {{Assessing XAI: Unveiling Evaluation Metrics for Local Explanation, Taxonomies, Key Concepts, and Practical Applications}},
  year = {2023},
  doi = {10.31224/2989},
}
Figure