Introduction
The Explainable AI (XAI) field is currently bogged down by conflicting goals and misconceptions about what explanations are actually for. Researchers, practitioners, and reviewers often have divergent, incompatible expectations for what an explanation should achieve. For instance, reviewers frequently demand user studies for new XAI methods, assuming all explanations must be tailored for laypeople to build "trust."
This position paper proposes dividing XAI into two distinct cultures: BLUE XAI (focused on human values, trust, and end-users) and RED XAI (focused on model validation, debugging, and developers). The authors argue that RED XAI is critically under-explored and outline key research challenges to advance model exploration and AI safety.
The Five Fallacies of XAI
The authors identify five major fallacies holding the field back:
- Interpretability is a binary concept (transparent vs. black-box).
- A single "silver bullet" XAI method exists.
- There is a "true" ground-truth explanation that XAI must approximate.
- User studies are the ultimate validation for all XAI methods.
- Explanations must always aim to increase end-user trust.
This confusion obscures the need for advanced, highly technical XAI tools meant strictly for AI developers. If every XAI tool must be simple enough for a layperson to understand, we severely limit our ability to debug and explore complex, high-stakes models.
The Two Cultures: RED and BLUE XAI
Inspired by cybersecurity's Red/Blue teaming and Leo Breiman's famous "Two Cultures" of statistical modeling, the authors propose splitting XAI research into two distinct tracks:
- BLUE XAI (Human-Values Oriented): Focuses on responsibility, legal issues, trust, and ethics. The audience is lay users, customers, or patients. The goal is to provide simple, easy-to-understand explanations during model inference to justify decisions.
- RED XAI (Model-Validation Oriented): Focuses on researching data, exploring models, and debugging. The audience is power users, model developers, and AI researchers. The goal is to provide actionable explanations that are strictly faithful to the model and data, often used during the training and auditing phases to question and fix the model.
Key Insights and Ideas
- The "Ground Truth" Fallacy: Evaluating XAI by comparing it to a human expert's "ground truth" (e.g., a doctor's tumor mask in medical imaging) can be misleading. A mismatch might not mean the XAI method is bad; it might reveal the model is using unexpected (but valid) features, or it might expose a critical flaw in the model itself.
- User Studies Aren't Always Necessary: Demanding user studies for every XAI paper stifles the development of complex, developer-focused debugging tools. A tool built to help an ML engineer find data leakage doesn't need to be validated by a non-technical crowd.
- The "Blind Men and the Elephant" Approach: No single XAI method tells the whole story. We need complementary explanations (multi-faceted XAI) that juxtapose different perspectives (e.g., feature importance alongside counterfactuals) to truly understand complex models.
- The Rashomon Set: Instead of looking for one perfect model, researchers should use XAI to explore the "Rashomon set"—a group of models with similar high performance but different internal logic. XAI can help pick the most robust or unbiased option from this set.
- XAI for Science: Explanations shouldn't just justify decisions; they should be used to discover new scientific truths. For example, analyzing the internal logic of a superhuman chess model or a drug-discovery model can teach humans new strategies and structural classes.
Future Directions for RED XAI
The current academic narrative overwhelmingly focuses on BLUE XAI concepts (trustworthiness, fairness, human alignment), leaving RED XAI topics (debugging, interactive exploration) vastly under-represented. The authors propose several promising directions for the future of RED XAI:
- Standardized Tooling: Developing benchmarks and tools specifically for model debugging (similar to how
scikit-learnstandardized general ML). - Interactive Exploration: Moving beyond static explanations to interactive, drill-down interfaces.
- Frontier Models: Applying RED XAI to foundation models (LLMs) to understand latent representations, edit facts, and study phenomena like "grokking."
- Expanding the Taxonomy: Exploring "Purple XAI" (methods useful to both developers and end-users, like SHAP) and "Yellow XAI" (frameworks for evaluating XAI methods themselves).
Summary updated