Good Explanations in Explainable Artificial Intelligence (XAI): Evidence from Human Explanatory Reasoning

Ruth M. J. ByrneIJCAI 2023 · DOI

TL;DR

This paper argues that to build truly user-centered AI explanations, we need to understand how humans naturally construct and interpret explanations. By drawing on cognitive science research, it highlights common human reasoning biases and tendencies that XAI systems must account for to be effective and trustworthy.

What Problem Does This Paper Address?

As AI decision-support systems become more prevalent across various sectors, a critical challenge is ensuring users trust and understand their decisions. The field of Explainable AI (XAI) aims to provide these explanations, but current approaches often overlook how humans actually reason. This paper addresses the gap between how AI systems generate explanations and how humans process them, arguing that a deeper understanding of human explanatory reasoning is crucial for developing XAI that is not just technically sound but also psychologically robust and genuinely useful to end-users. Without this, explanations might create an "illusion of understanding" without actually improving comprehension, leading to misplaced trust or misunderstanding.

Methods

This paper is a survey of existing cognitive science research on human explanatory reasoning. It doesn't introduce new algorithms or experimental methods but rather synthesizes findings from psychological studies to inform XAI. The "methods" discussed are those used in cognitive science research, such as:

  • Experimental studies: Participants are presented with scenarios (e.g., how devices work, causal relationships) and asked to explain them, make inferences, or judge their understanding.
  • Diagnostic tasks: Users are asked to predict AI system decisions to objectively measure their understanding.
  • Meta-cognitive judgments: Self-report measures of satisfaction, trust, and perceived understanding are used, but their limitations are also discussed.
  • Mental model theory: This theoretical framework is used to explain how people represent causal relationships and make inferences.

Datasets

Not discussed in the paper. The paper focuses on synthesizing existing psychological research rather than presenting new empirical studies with datasets.

Key Insights and Ideas

  • The Illusion of Explanatory Depth: People often overestimate their understanding of how things work. The act of trying to explain something can reveal their knowledge gaps, leading to a more accurate self-assessment. This implies that simply asking users if they understand an AI explanation isn't enough; objective measures are needed.
  • Intuitive vs. Reflective Explanations: Humans often start with fast, intuitive explanations and then refine them with slower, deliberative thinking. XAI systems need to consider whether their explanations align with these dual processing modes, potentially catering to both initial intuitions and deeper reflection.
  • Explanatory Stances: People adopt different perspectives (e.g., mechanical, design, intentional) when explaining things. XAI systems might need to offer explanations tailored to the user's preferred stance or the nature of the AI's decision.
  • Inference Suppression and Causal Discounting: When presented with enabling conditions or alternative causes, people tend to suppress inferences or discount certain causes. This means XAI explanations that focus on a single cause might be less effective if users are aware of other contributing factors.
  • Simplicity Preference: Humans generally prefer simpler explanations, especially common-cause explanations. However, this preference can be overridden by evidence or when a complex explanation better fits the data. XAI needs to balance simplicity with accuracy.
  • Dissociation between Subjective and Objective Measures: Users might feel they understand an AI system better after a certain type of explanation (e.g., counterfactuals), but their actual predictive accuracy might not improve significantly compared to other explanation types. This highlights the danger of relying solely on user satisfaction metrics.

Key Results

The paper doesn't present quantitative results in terms of accuracy or performance metrics for XAI systems. Instead, it synthesizes qualitative findings from cognitive science research, such as:

  • Experimental participants consistently show an "illusion of explanatory depth" when asked to explain complex devices or AI systems.
  • Studies demonstrate that people make different inferences from causal versus counterfactual explanations.
  • Research indicates that people tend to suppress inferences when enabling conditions are present or discount causes when alternatives exist.
  • Participants show a preference for simpler, common-cause explanations, but this can be context-dependent.
  • User studies have shown a dissociation where users report higher satisfaction with counterfactual explanations, even if their objective understanding (measured by prediction accuracy) is similar to that from causal explanations.

Limitations and Future Work

  • Scope of Survey: The paper acknowledges that it can only highlight a limited number of psychological phenomena and necessarily omits other relevant areas, such as graphical explanations or the impact of social and cultural factors.
  • AI Bias: The authors note the critical concern that AI systems themselves can introduce biases into decisions and explanations, which current XAI strategies may not fully address.
  • Future Work: The paper strongly implies that future XAI development should:
  • Employ rigorous experimental methods with objective measures (like prediction accuracy) to evaluate explanation efficacy, rather than relying solely on self-report satisfaction.
  • Develop aids to help users calibrate their self-understanding of AI systems, perhaps through interactive explanations or diagnostic tasks.
  • Design explanations that account for human tendencies like inference suppression and causal discounting, potentially by acknowledging alternative causes or enabling conditions.
  • Consider the user's preferred "explanatory stance" when generating explanations.
  • Balance the preference for simplicity with the need for accurate and comprehensive explanations.

Summary updated

BibTeX
@inproceedings{byrne2023good,
  author = {Ruth M. J. Byrne},
  title = {{Good Explanations in Explainable Artificial Intelligence (XAI): Evidence from Human Explanatory Reasoning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/733},
}
Figure