TL;DR
This paper surveys cognitive science literature to understand what makes AI explanations effective for humans. It argues that current XAI efforts often overlook the diverse needs of different stakeholders and proposes a user-centered approach to bridge the gap between AI research and practical adoption.
What Problem Does This Paper Address?
As AI systems become more prevalent in critical domains like healthcare and finance, the demand for explainability (XAI) is growing. However, many current XAI efforts focus on technical explanations suitable for AI scientists or domain experts, neglecting the needs of other stakeholders like business executives or consumers. This gap leads to explanations that are often unhelpful or even confusing for the intended audience. The paper aims to address this by exploring what constitutes a good explanation from a human-centric, cognitive science perspective, considering the purpose, structure, and characteristics of effective explanations for various users.
Methods
The paper is a survey of existing literature from the cognitive sciences. It systematically reviews research to answer three core questions:
- What is an explanation? It explores various theoretical frameworks for explanations, including deductive proofs, causal patterns (common cause, common effect, linear chains, causal homeostasis), mental models, and stances (mechanical, illustrative, intentional).
- What are explanations for? It delves into the various functions explanations serve for humans, such as building trust, aiding troubleshooting, educating users, and enabling action.
- What are the characteristics of good and bad explanations? It identifies key features that make explanations effective or ineffective, such as simplicity, robustness, coherence, relevance, and self-evidencing properties, while also highlighting pitfalls like circularity and excessive length.
Datasets
Not discussed in the paper. The paper is a survey of cognitive science literature and does not present new empirical studies or datasets.
Key Insights and Ideas
- Explanations are not one-size-fits-all: The effectiveness of an explanation heavily depends on the specific human-in-the-loop, their background knowledge, and the purpose they seek to achieve with the explanation.
- Beyond technical correctness: While AI scientists might appreciate rule-based or deductive explanations, a loan applicant needs information on how to improve their chances for future loans, highlighting the need for explanations to serve behavioral purposes.
- Cognitive science offers a roadmap: Understanding how humans naturally explain and understand phenomena provides valuable insights into designing more effective XAI systems.
- The "why" is crucial: Simply stating a correlation or a rule is often insufficient. Good explanations help bridge the gap between knowing what happened and understanding why it happened.
- User-centered design is paramount: To bridge the gap between XAI research and practice, a shift towards user-centered design, considering the specific needs and contexts of different stakeholders, is essential.
Key Results
The paper does not present quantitative results as it is a survey. Its key "results" are the synthesized insights from the cognitive science literature, which include:
- A categorization of explanation theories (deductive proofs, causal patterns, mental models, stances) and their respective strengths and weaknesses.
- An enumeration of the diverse functions of explanations (trust, troubleshooting, education, action, justification, aesthetics, communication).
- A list of characteristics that define good (simple, robust, addresses knowledge gaps, self-evidencing) and bad (circular, irrelevant, incoherent, lengthy) explanations.
- Recommendations for XAI designers, emphasizing user-centered approaches, domain-specific knowledge incorporation, understanding explanation context, and creating new datasets and algorithms tailored for user-friendly explanations.
Limitations and Future Work
Limitations: * The paper acknowledges that some explanation stances, particularly intentional ones, can be subjective and difficult to predict. * It notes that some causal patterns might be domain-specific, and the underlying causal structures can be influenced by human subjectivity. * Explanations based on neural activation are perceived as black-box models, offering little insight into internal functionalities.
Future Work: The paper strongly advocates for:
- Adopting a user-centered approach in generating explanations, tailoring them to specific user types and their needs.
- Incorporating domain-specific knowledge into AI models to enhance explainability and reduce inconsistencies.
- Thoroughly understanding the context of an explanation (who, what purpose, etc.) to personalize it.
- Creating new datasets that better represent user-friendly explanations and developing robust algorithms for smaller datasets.
- Evaluating XAI systems not just on algorithmic performance but also on how well explanations satisfy the user's purpose, incorporating user-experience studies.
Summary updated