TL;DR
Explainable AI (XAI) is suffering from a "crisis of identity" caused by trying to satisfy too many conflicting goals at once. This position paper proposes splitting the field into two distinct cultures: BLUE XAI (human-centric justification) and RED XAI (model-centric debugging), arguing that we must prioritize "questioning" models over simply "justifying" them to ensure AI safety and scientific progress.
What Problem Does This Paper Address?
The authors argue that XAI research is currently slowed down by a lack of clear objectives. Researchers often conflate the need to provide a "right to explanation" for lay users (e.g., a bank customer denied a loan) with the need for developers to debug complex models.
This confusion leads to several "fallacies," such as the belief that a single "silver bullet" XAI method exists or that user studies are the only valid way to evaluate an explanation. By trying to make every explanation simple enough for a layperson, the field risks neglecting the advanced, technical tools needed by engineers to find "unknown unknowns" and fix bugs in high-stakes AI systems.
Methods
This is a position paper that introduces a new conceptual framework for the XAI community, inspired by Leo Breiman’s seminal "Two Cultures" of statistical modeling. The authors categorize XAI into:
- BLUE XAI (Human-Values Oriented): Focused on trust, fairness, and ethics. The audience is the lay user, and the goal is often to provide a simple, understandable justification for a model's decision.
- RED XAI (Model-Validation Oriented): Focused on research, exploration, and debugging. The audience is the "power user" (developer or researcher), and the goal is to faithfully represent model behavior, even if the resulting explanation is complex.
The paper also advocates for a shift from "I generation" explanations (single-aspect, one-shot) to "II generation" explanations, which are interactive and multifaceted.
Datasets
Not discussed in the paper. As a position paper, it focuses on theoretical frameworks rather than empirical results on specific datasets. However, it does reference the FICO Explainable Machine Learning Challenge as a successful example of a benchmark that spurred innovation in model debugging.
Key Insights and Ideas
- The "Silver Bullet" Fallacy: There is no single best XAI method. The authors suggest viewing XAI techniques as different "magnification lenses." Just as a biologist needs both a light microscope and an electron microscope, an AI developer needs multiple, complementary views to understand a model.
- The Rashomon Perspective: Instead of explaining one "best" model, we should explore the "Rashomon set"—a group of different models that all achieve similar accuracy but arrive at their predictions through different logic. Comparing these models helps identify which relationships in the data are robust and which are artifacts of a specific architecture.
- Interactive Exploration: Static feature importance scores are often insufficient. The authors point to tools like the "What-If Tool" as the future of RED XAI, where developers can interactively probe model behavior.
Key Results
The paper’s primary contribution is a diagnostic roadmap for the field:
- Formalized Taxonomy: It provides a clear vocabulary (RED vs. BLUE) to help reviewers and researchers align their expectations.
- Identified Research Gaps: The authors show that while BLUE XAI dominates current literature, RED XAI is under-explored and holds the key to AI safety.
- Stakeholder Mapping: Figure 4 illustrates how different stakeholders (patients vs. AI researchers) ask fundamentally different questions that require different XAI approaches.
Limitations and Future Work
- The "Secret Sauce" Problem: The authors acknowledge that RED XAI methods—which provide a competitive advantage in building better models—might be kept proprietary by companies, potentially slowing down public academic research.
- Beyond Two Colors: While the paper focuses on RED and BLUE, it briefly mentions "Purple XAI" (methods like SHAP that sit in the middle) and "Yellow XAI" (focused on frameworks and benchmarks).
- Future Directions: The authors call for a new "XAI-centric" curriculum in Machine Learning education and the development of automated "checks" that can systematically integrate different explanatory views to summarize model behavior.
Summary updated