From Anecdotal Evidence to Quantitative Evaluation Methods

Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, Christin SeifertUniversity of Twente Research Information (Pure Portal) 2025 · OpenReview

TL;DR

This paper addresses the critical need for robust evaluation methods in Explainable AI (XAI). It proposes a comprehensive framework of 12 properties (Co-12) for assessing explanation quality and systematically reviews over 300 XAI papers to catalog existing quantitative evaluation techniques, highlighting the current over-reliance on anecdotal evidence.

What Problem Does This Paper Address?

As AI models, particularly deep learning systems, become increasingly complex and powerful, understanding why they make certain decisions (explainability) is crucial for trust, debugging, and fairness. However, evaluating the quality of these explanations has been a significant challenge. Researchers often rely on subjective, anecdotal evidence – showing a few convincing examples – rather than rigorous, quantitative metrics. This lack of standardized evaluation hinders progress, making it difficult to compare different XAI methods or ensure their reliability. The paper aims to bridge this gap by providing a structured approach to evaluating explanations and a comprehensive overview of existing quantitative methods.

Methods

The authors conducted a systematic literature review of over 300 XAI papers published between 2014 and 2020 across major AI/ML conferences. Their methodology involved:

  1. Categorization of XAI Methods: Papers were categorized based on six dimensions: type of data, type of explanation, type of problem, type of predictive model, type of task, and type of method used to explain.
  2. Development of the Co-12 Properties: Based on conceptual literature, they identified 12 key properties that define explanation quality, such as Correctness (faithfulness to the model), Completeness (how much of the model's behavior is explained), Compactness (size of the explanation), and Coherence (alignment with prior beliefs). These properties are grouped into Content, Presentation, and User dimensions.
  3. Cataloging Quantitative Evaluation Methods: The review focused on identifying and describing quantitative, functionally-grounded evaluation methods (i.e., those not requiring user studies) that map to the Co-12 properties.

Datasets

The paper does not introduce new datasets. Instead, it analyzes existing research that utilizes various datasets relevant to the XAI methods they discuss. The focus is on the evaluation methodologies applied to these datasets, not on the datasets themselves.

Key Insights and Ideas

  • Explainability is Multi-Faceted: The paper strongly argues against viewing explainability as a simple binary property. The proposed Co-12 properties highlight that a good explanation needs to satisfy multiple criteria simultaneously, and trade-offs are often necessary.
  • Anecdotal Evidence is Prevalent but Insufficient: A significant finding is that one in three papers rely exclusively on anecdotal evidence for evaluation, underscoring the need for more rigorous methods.
  • Quantitative Evaluation is Maturing but Underutilized: While the number of papers employing quantitative evaluation has increased, it still lags behind the overall growth of XAI research. The paper provides a valuable resource for researchers to adopt these methods.
  • Coherence and Output-Completeness are Popular (but not exhaustive): Among the Co-12 properties, Coherence and Output-completeness are the most frequently evaluated quantitatively, suggesting a focus on how explanations align with user expectations and how well they capture model outputs.
  • The Need for Standardization: The paper advocates for establishing standardized evaluation practices and terminology to facilitate better comparison and benchmarking of XAI methods.

Key Results

  • Evaluation Practices:
    • 33% of papers introducing an XAI method evaluated it only with anecdotal evidence.
  • 58% applied quantitative evaluation methods.
  • 22% included user studies.
  • Co-12 Properties: The paper identifies and defines 12 key properties for explanation quality, providing a structured framework for evaluation.
  • Catalog of Quantitative Methods: The paper presents a comprehensive overview of quantitative evaluation methods, categorized by the Co-12 properties they assess, serving as a practical guide for researchers.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the selection of venues might influence the statistics on quantitative evaluation usage.
  • The paper notes that some XAI methods are presented as inherently interpretable (white-box models), and thus not explicitly evaluated for interpretability, though the authors argue they should be held to similar standards.
  • The authors mention that for some papers, the inclusion criteria were checked by two reviewers, and agreement metrics might be biased towards lower scores due to the perceived difficulty of the papers.
  • Future Work:
    • The authors suggest developing new evaluation methods for Co-12 properties that are currently under-addressed.
  • They propose creating variants of existing methods to suit different data types and explanations.
  • A significant research opportunity lies in incorporating quantitative evaluation methods directly into the model training process to optimize the "accuracy-interpretability trade-off" rather than just evaluating it post-hoc.
  • The paper makes its annotated dataset publicly available, encouraging further in-depth research on specific subtopics.

Summary updated

Abstract

The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes, also raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practice of more than 300 papers published in the last 7 years at major AI and ML conferences that introduce an XAI method. We find that 1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with users. We also contribute to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. This systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark and compare new and existing XAI methods. This also opens up opportunities to include quantitative metrics as optimization criteria during model training in order to optimize for accuracy and interpretability simultaneously.
BibTeX
@inproceedings{nauta2025from,
  author = {Meike Nauta and Jan Trienes and Shreyasi Pathak and Elisa Nguyen and Michelle Peters and Yasmin Schmitt and Jörg Schlötterer and Maurice van Keulen and Christin Seifert},
  title = {{From Anecdotal Evidence to Quantitative Evaluation Methods}},
  booktitle = {University of Twente Research Information (Pure Portal)},
  year = {2025},
  url = {https://openreview.net/forum?id=SpNkdLmmwb},
}
Figure