Explain to Question not to Justify

Przemyslaw Biecek, Wojciech SamekarXiv 2024 · OpenReview

TL;DR

Explainable AI (XAI) is suffering from a "crisis of identity" caused by trying to satisfy too many conflicting goals at once. This position paper proposes splitting the field into two distinct cultures: BLUE XAI (human-centric justification) and RED XAI (model-centric debugging), arguing that we must prioritize "questioning" models over simply "justifying" them to ensure AI safety and scientific progress.

What Problem Does This Paper Address?

The authors argue that XAI research is currently slowed down by a lack of clear objectives. Researchers often conflate the need to provide a "right to explanation" for lay users (e.g., a bank customer denied a loan) with the need for developers to debug complex models.

This confusion leads to several "fallacies," such as the belief that a single "silver bullet" XAI method exists or that user studies are the only valid way to evaluate an explanation. By trying to make every explanation simple enough for a layperson, the field risks neglecting the advanced, technical tools needed by engineers to find "unknown unknowns" and fix bugs in high-stakes AI systems.

Methods

This is a position paper that introduces a new conceptual framework for the XAI community, inspired by Leo Breiman’s seminal "Two Cultures" of statistical modeling. The authors categorize XAI into:

  • BLUE XAI (Human-Values Oriented): Focused on trust, fairness, and ethics. The audience is the lay user, and the goal is often to provide a simple, understandable justification for a model's decision.
  • RED XAI (Model-Validation Oriented): Focused on research, exploration, and debugging. The audience is the "power user" (developer or researcher), and the goal is to faithfully represent model behavior, even if the resulting explanation is complex.

The paper also advocates for a shift from "I generation" explanations (single-aspect, one-shot) to "II generation" explanations, which are interactive and multifaceted.

A summary of the paper's main claims, contrasting the two cultures and identifying new challenges.

Datasets

Not discussed in the paper. As a position paper, it focuses on theoretical frameworks rather than empirical results on specific datasets. However, it does reference the FICO Explainable Machine Learning Challenge as a successful example of a benchmark that spurred innovation in model debugging.

Key Insights and Ideas

  • The "Silver Bullet" Fallacy: There is no single best XAI method. The authors suggest viewing XAI techniques as different "magnification lenses." Just as a biologist needs both a light microscope and an electron microscope, an AI developer needs multiple, complementary views to understand a model.
  • The Rashomon Perspective: Instead of explaining one "best" model, we should explore the "Rashomon set"—a group of different models that all achieve similar accuracy but arrive at their predictions through different logic. Comparing these models helps identify which relationships in the data are robust and which are artifacts of a specific architecture.
  • Interactive Exploration: Static feature importance scores are often insufficient. The authors point to tools like the "What-If Tool" as the future of RED XAI, where developers can interactively probe model behavior.
Figure 2. Schematic diagram of multifaceted explanations. Many explanation techniques present one aspect of how models work. Multifaceted explanations show several complementary aspects simultaneously. Figure adapted from (Baniecki et al., 2023b)

Key Results

The paper’s primary contribution is a diagnostic roadmap for the field:

  • Formalized Taxonomy: It provides a clear vocabulary (RED vs. BLUE) to help reviewers and researchers align their expectations.
  • Identified Research Gaps: The authors show that while BLUE XAI dominates current literature, RED XAI is under-explored and holds the key to AI safety.
  • Stakeholder Mapping: Figure 4 illustrates how different stakeholders (patients vs. AI researchers) ask fundamentally different questions that require different XAI approaches.
Stakeholder questions for a hypothetical COVID-19 risk model, illustrating the split between BLUE and RED perspectives.

Limitations and Future Work

  • The "Secret Sauce" Problem: The authors acknowledge that RED XAI methods—which provide a competitive advantage in building better models—might be kept proprietary by companies, potentially slowing down public academic research.
  • Beyond Two Colors: While the paper focuses on RED and BLUE, it briefly mentions "Purple XAI" (methods like SHAP that sit in the middle) and "Yellow XAI" (focused on frameworks and benchmarks).
  • Future Directions: The authors call for a new "XAI-centric" curriculum in Machine Learning education and the development of automated "checks" that can systematically integrate different explanatory views to summarize model behavior.

Summary updated

Abstract

Explainable Artificial Intelligence (XAI) is a young but very promising field of research. Unfortunately, the progress in this field is currently slowed down by divergent and incompatible goals. We separate various threads tangled within the area of XAI into two complementary cultures of human/value-oriented explanations (BLUE XAI) and model/validation-oriented explanations (RED XAI). This position paper argues that the area of RED XAI is currently under-explored, i.e., more methods for explainability are desperately needed to question models (e.g., extract knowledge from well-performing models as well as spotting and fixing bugs in faulty models), and the area of RED XAI hides great opportunities and potential for important research necessary to ensure the safety of AI systems. We conclude this paper by presenting promising challenges in this area.
BibTeX
@misc{biecek2024explain,
  author = {Przemyslaw Biecek and Wojciech Samek},
  title = {{Explain to Question not to Justify}},
  year = {2024},
  eprint = {2402.13914},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.13914},
}
Figure