Position: Explain to Question not to Justify

Przemyslaw Biecek, Wojciech Samek

TL;DR

The field of Explainable AI (XAI) is currently facing an "identity crisis" due to conflicting goals. This paper argues for a clearer distinction between two XAI cultures: "BLUE XAI" focused on human values and trust, and "RED XAI" focused on model validation and debugging. The authors contend that RED XAI is under-explored and holds significant potential for advancing AI safety and understanding.

What Problem Does This Paper Address?

The progress in Explainable Artificial Intelligence (XAI) is being hampered by divergent and often incompatible goals. Researchers are pulled in different directions, leading to confusion about the purpose and evaluation of XAI methods. This paper identifies a core issue: the field is often framed around user-centric explanations aimed at building trust ("BLUE XAI"), while neglecting the equally, if not more, critical area of using XAI for rigorous model validation, debugging, and knowledge extraction ("RED XAI"). This imbalance hinders the development of truly robust and safe AI systems.

Methods

This paper is primarily a position paper and a conceptual framework, rather than a method-heavy paper. The authors' "method" involves:

  • Deconstructing Fallacies: Identifying and challenging five common misconceptions in XAI research, such as interpretability being a binary concept or the existence of a single "silver bullet" explanation method.
  • Cultural Dichotomy: Proposing a framework that separates XAI research into two distinct cultures:
  • BLUE XAI (Human Values Oriented Explanations): Focused on user trust, fairness, ethics, and legal aspects. Explanations are primarily for end-users.
  • RED XAI (Model Validation Oriented Explanations): Focused on model developers, debugging, exploration, and understanding model behavior. Explanations are for AI experts.
  • Identifying New Challenges: Based on the RED XAI framework, the authors outline promising new research directions and challenges.

Datasets

Not discussed in the paper. The paper focuses on conceptual framing and challenges rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • XAI is not a Monolith: The field is not a single entity but rather a collection of diverse goals and approaches.
  • The "Crisis of Identity": The current confusion in XAI stems from a lack of clear objectives and a tendency to conflate different needs.
  • BLUE vs. RED XAI: This dichotomy is crucial. BLUE XAI aims to make AI systems understandable and trustworthy for users, while RED XAI aims to make AI systems understandable and debuggable for developers.
  • RED XAI is Under-Explored: While BLUE XAI receives significant attention (especially in surveys), RED XAI offers fertile ground for research in areas like debugging, knowledge extraction, and improving model robustness.
  • Questioning, Not Just Justifying: The paper advocates for XAI methods that help question models (e.g., find bugs, extract knowledge) rather than solely justifying their decisions.
  • Fallacies Hinder Progress: Common assumptions about XAI (e.g., user studies as the ultimate validation, binary interpretability) are misleading and can slow down research.
  • Multifaceted Explanations: Complex models require multiple, complementary explanations, akin to blind men describing an elephant, to gain a holistic understanding.

Key Results

This paper does not present quantitative results from experiments. Its primary contribution is a conceptual reframing of the XAI landscape and the identification of new research avenues. The "results" are the insights and proposed challenges.

Limitations and Future Work

  • Oversimplification: The authors acknowledge that the RED/BLUE dichotomy might be an oversimplification and that some applications might blend aspects of both. They suggest the possibility of "Purple XAI" for methods usable by both user groups.
  • Need for Benchmarks and Tools: The paper highlights a significant need for standardized benchmarks, tools, and best practices for RED XAI to facilitate progress and comparison.
  • Explorer Mindset: Future work should encourage an "explorer mindset" in RED XAI, embracing the discovery of "unknown unknowns" in model behavior.
  • Multiple Models and Rashomon Explanations: Investigating sets of models with similar performance but different behaviors (Rashomon sets) is a promising avenue for deeper understanding.
  • XAI for Science: The paper suggests that XAI can be a powerful tool for generating and validating scientific hypotheses, opening up new research frontiers.
  • Education Reform: The authors believe there's a need for more holistic XAI and debugging-centric education programs for AI professionals.

Summary updated

BibTeX
@inproceedings{biecek2024position,
  author = {Przemyslaw Biecek and Wojciech Samek},
  title = {{Position: Explain to Question not to Justify}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=ooikIHLHCs},
}
Figure