TL;DR
This paper addresses the critical need for robust evaluation methods in Explainable AI (XAI). It proposes a comprehensive framework of 12 properties (Co-12) for assessing explanation quality and systematically reviews over 300 XAI papers to catalog existing quantitative evaluation techniques, highlighting the current over-reliance on anecdotal evidence.
What Problem Does This Paper Address?
As AI models, particularly deep learning systems, become increasingly complex and powerful, understanding why they make certain decisions (explainability) is crucial for trust, debugging, and fairness. However, evaluating the quality of these explanations has been a significant challenge. Researchers often rely on subjective, anecdotal evidence – showing a few convincing examples – rather than rigorous, quantitative metrics. This lack of standardized evaluation hinders progress, making it difficult to compare different XAI methods or ensure their reliability. The paper aims to bridge this gap by providing a structured approach to evaluating explanations and a comprehensive overview of existing quantitative methods.
Methods
The authors conducted a systematic literature review of over 300 XAI papers published between 2014 and 2020 across major AI/ML conferences. Their methodology involved:
- Categorization of XAI Methods: Papers were categorized based on six dimensions: type of data, type of explanation, type of problem, type of predictive model, type of task, and type of method used to explain.
- Development of the Co-12 Properties: Based on conceptual literature, they identified 12 key properties that define explanation quality, such as Correctness (faithfulness to the model), Completeness (how much of the model's behavior is explained), Compactness (size of the explanation), and Coherence (alignment with prior beliefs). These properties are grouped into Content, Presentation, and User dimensions.
- Cataloging Quantitative Evaluation Methods: The review focused on identifying and describing quantitative, functionally-grounded evaluation methods (i.e., those not requiring user studies) that map to the Co-12 properties.
Datasets
The paper does not introduce new datasets. Instead, it analyzes existing research that utilizes various datasets relevant to the XAI methods they discuss. The focus is on the evaluation methodologies applied to these datasets, not on the datasets themselves.
Key Insights and Ideas
- Explainability is Multi-Faceted: The paper strongly argues against viewing explainability as a simple binary property. The proposed Co-12 properties highlight that a good explanation needs to satisfy multiple criteria simultaneously, and trade-offs are often necessary.
- Anecdotal Evidence is Prevalent but Insufficient: A significant finding is that one in three papers rely exclusively on anecdotal evidence for evaluation, underscoring the need for more rigorous methods.
- Quantitative Evaluation is Maturing but Underutilized: While the number of papers employing quantitative evaluation has increased, it still lags behind the overall growth of XAI research. The paper provides a valuable resource for researchers to adopt these methods.
- Coherence and Output-Completeness are Popular (but not exhaustive): Among the Co-12 properties, Coherence and Output-completeness are the most frequently evaluated quantitatively, suggesting a focus on how explanations align with user expectations and how well they capture model outputs.
- The Need for Standardization: The paper advocates for establishing standardized evaluation practices and terminology to facilitate better comparison and benchmarking of XAI methods.
Key Results
- Evaluation Practices:
- 33% of papers introducing an XAI method evaluated it only with anecdotal evidence.
- 58% applied quantitative evaluation methods.
- 22% included user studies.
- Co-12 Properties: The paper identifies and defines 12 key properties for explanation quality, providing a structured framework for evaluation.
- Catalog of Quantitative Methods: The paper presents a comprehensive overview of quantitative evaluation methods, categorized by the Co-12 properties they assess, serving as a practical guide for researchers.
Limitations and Future Work
- Limitations:
- The study acknowledges that the selection of venues might influence the statistics on quantitative evaluation usage.
- The paper notes that some XAI methods are presented as inherently interpretable (white-box models), and thus not explicitly evaluated for interpretability, though the authors argue they should be held to similar standards.
- The authors mention that for some papers, the inclusion criteria were checked by two reviewers, and agreement metrics might be biased towards lower scores due to the perceived difficulty of the papers.
- Future Work:
- The authors suggest developing new evaluation methods for Co-12 properties that are currently under-addressed.
- They propose creating variants of existing methods to suit different data types and explanations.
- A significant research opportunity lies in incorporating quantitative evaluation methods directly into the model training process to optimize the "accuracy-interpretability trade-off" rather than just evaluating it post-hoc.
- The paper makes its annotated dataset publicly available, encouraging further in-depth research on specific subtopics.
Summary updated