Abstract
Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs, objectives, and possibilities have emerged for explainability (XAI). In this work, we elaborate on why XAI has gained importance with the rise of GenAI and its challenges for explainability research. We also unveil novel and emerging desiderata that explanations should fulfill, covering aspects such as verifiability, interactivity, security, and cost. To this end, we focus on surveying existing works. Furthermore, we provide a taxonomy of relevant dimensions that allows us to better characterize existing XAI mechanisms and methods for GenAI. We discuss different avenues to ensure XAI, from training data to prompting. Our paper offers a short but concise technical background of GenAI for non-technical readers, focusing on text and images to better understand novel or adapted XAI techniques for GenAI. However, due to the vast array of works on GenAI, we decided to forego detailed aspects of XAI related to evaluation and usage of explanations. As such, the manuscript interests both technically oriented people and other disciplines, such as social scientists and information systems researchers. Our research roadmap provides more than ten directions for future investigation.
The Complexity Gap in Generative AI
Traditional Explainable AI (XAI) was primarily designed for "discriminative" models—systems that classify inputs, such as identifying whether an image contains a cat or a dog. In these scenarios, an explanation typically justifies a single discrete output.
Generative AI (GenAI) fundamentally changes this paradigm. When a Large Language Model (LLM) generates a multi-page essay or a Diffusion model creates a high-fidelity image, the system is making millions of interconnected decisions simultaneously. This creates a massive "complexity gap" characterized by three main hurdles:
- Output Complexity: Explaining the specific nuance of a generated sentence or a particular artistic style in an image is far more difficult than explaining a classification label.
- The Access Barrier: Many state-of-the-art models are closed-source, preventing researchers from performing "white-box" inspections of internal weights.
- The Hallucination Trap: GenAI can produce fluent but factually incorrect content. If a model provides a self-explanation that is itself a hallucination, it risks creating a dangerous cycle of misplaced user trust.
As shown in the figure above, the transition from single unimodal models to interconnected systems involving orchestration applications and external data sources significantly increases the surface area that requires explanation.
A Taxonomy for GenXAI
To address these challenges, this paper synthesizes the current landscape into a structured taxonomy based on several critical dimensions:
- Foundational Sources: Where does the explanation originate? It could stem from the model's weights, the training data, the specific prompt, or the optimization process itself.
- Technique Categories:
- Feature Attribution: Assigning relevance scores to input tokens or pixels (e.g., SHAP, Grad-CAM).
- Sample-based: Explaining outputs by referencing similar training examples or counterfactual scenarios.
- Probing-based: Using diagnostic classifiers to identify what latent knowledge (e.g., factual or grammatical) is stored in internal layers.
- Mechanistic Interpretability: Reverse-engineering internal circuits into human-understandable algorithms.
- Dynamics: Distinguishing between static, one-off explanations and interactive, user-driven dialogues.
Emerging Desiderata and the Economics of XAI
The shift to GenAI introduces new requirements for what constitutes a "good" explanation:
- Attribute Scope: Rather than explaining the entire output, we need tools to explain specific facets (e.g., "Why is this AI-generated person wearing a blue hat?").
- Lineage: In light of copyright and data integrity concerns, the ability to track a generated output back to specific training data points is becoming essential.
- The Economics of XAI: Explanations are not computationally free. They consume significant resources and can potentially leak intellectual property or expose security vulnerabilities, such as providing a roadmap for prompt injection attacks.
A Roadmap for Future Research
Conceptual framework of the GenXAI survey and its progression toward a research agenda. Source, p. 3
The paper concludes with a 12-point research roadmap designed to guide the next generation of XAI research. Key priorities include the development of Multimodal GenXAI (explaining video and 3D generation), improving Real-time Interactivity, and establishing Verifiability frameworks to ensure that explanations are not just plausible, but technically faithful to the model's internal logic. By moving beyond simple feature importance toward holistic system understanding, we can ensure that the "black box" of creation becomes a transparent tool for human-AI collaboration.
Summary updated