Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda

Johannes Schneider

Introduction

As artificial intelligence transitions from pattern recognition to the generation of complex, high-dimensional artifacts, the field of Explainable AI (XAI) faces a paradigm shift. Traditional XAI was built for classifiers—models that output simple labels (e.g., "Is this a cat?"). In contrast, Generative AI (GenAI) produces high-dimensional artifacts like long-form text, images, and code, where every single "bit" of the output might require its own explanation. This paper formalizes the field of Explainable Generative AI (GenXAI), providing a comprehensive survey, a new taxonomy, and a roadmap for future research.

Conceptual framework and structure of the GenXAI survey.

The Challenge of Generative AI

The author identifies a critical gap: existing XAI frameworks do not fully account for the unique challenges of modern foundation models. Unlike simple classifiers, GenAI models like GPT-4 are often "black boxes" accessed via APIs, involve interactive prompt engineering, and are prone to "hallucinations."

Explaining a single classification is straightforward; explaining a 1024x1024 image involves explaining millions of pixels and their relationships. This "Million-Bit" problem necessitates Dynamic Explanations that automatically focus on the most surprising or impactful parts of the output. Furthermore, there is a dire need for verifiability to combat hallucinations and ensure the security of these generative systems.

A Taxonomy for GenXAI

This paper is a meta-survey and conceptual review that synthesizes a vast body of recent literature to build a novel taxonomy for GenXAI. The author categorizes XAI techniques into four established groups, adapted for the generative context:

  1. Feature Attribution: Assigning relevance to specific input tokens or pixels (e.g., Grad-CAM, SHAP).
  2. Sample-based: Explaining by looking at similar training data or counterfactuals (e.g., "What's the smallest change to this prompt that would change the output?").
  3. Probing-based: Training "probe" classifiers on internal model activations to see what the model "knows" (e.g., syntax vs. semantics).
  4. Mechanistic Interpretability: Reverse-engineering the "circuits" of neurons to understand how specific behaviors emerge from the model's architecture.
Internal components of a Transformer model, such as Multi-Head Attention, which are targets for mechanistic interpretability.

The paper also introduces novel dimensions for categorization, such as the foundational source of the explanation—whether it is derived from the training data, the model architecture, the optimization process, or the prompt itself.

Key Desiderata and Insights

The paper identifies several new criteria and insights essential for the GenAI era:

  • The Rise of Self-Explanation: Unlike older models, LLMs can "self-explain" (e.g., Chain-of-Thought prompting). While this makes AI more accessible, these explanations are subject to the same hallucinations as the primary output.
  • XAI as a Tool for Users: In the pre-GenAI era, XAI was primarily for model debugging. Now, it is a requirement for "prompt engineering," helping everyday users understand how to tweak their inputs for better results.
  • New Desiderata: The author proposes criteria such as Verifiability (can the user check if the explanation is true?), Lineage (can we trace a fact back to a training document?), and Security (ensuring explanations don't reveal proprietary secrets or enable jailbreaks).

A Roadmap for Future Research

The primary contribution of the paper is a structured Research Agenda consisting of 12 directions. Key highlights include:

  • Multimodal GenXAI: Addressing the lack of techniques that can explain an image using text (or vice versa) in a unified way.
  • Interactive XAI: Moving from static heatmaps to a "dialogue" where users can ask the AI specific questions about its decisions.
  • The Economics of XAI: Balancing the significant computational costs of generating explanations against the value they provide.

Limitations and Future Work

The author notes that the survey focuses primarily on text and images, pointing to video, 3D content generation, and AI "actions" as the next frontiers. Additionally, measuring "faithfulness" (whether an explanation accurately reflects the model's inner workings) remains an open problem. Finally, the "black-box" nature of commercial APIs remains a major hurdle, as many powerful XAI techniques require "white-box" access to internal gradients and weights.

Summary updated

Abstract

Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs, objectives, and possibilities have emerged for explainability (XAI). In this work, we elaborate on why XAI has gained importance with the rise of GenAI and its challenges for explainability research. We also unveil novel and emerging desiderata that explanations should fulfill, covering aspects such as verifiability, interactivity, security, and cost. To this end, we focus on surveying existing works. Furthermore, we provide a taxonomy of relevant dimensions that allows us to better characterize existing XAI mechanisms and methods for GenAI. We discuss different avenues to ensure XAI, from training data to prompting. Our paper offers a short but concise technical background of GenAI for non-technical readers, focusing on text and images to better understand novel or adapted XAI techniques for GenAI. However, due to the vast array of works on GenAI, we decided to forego detailed aspects of XAI related to evaluation and usage of explanations. As such, the manuscript interests both technically oriented people and other disciplines, such as social scientists and information systems researchers. Our research roadmap provides more than ten directions for future investigation.
BibTeX
@article{schneider2024explainable,
  author = {Johannes Schneider},
  title = {{Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda}},
  journal = {Artif. Intell. Rev.},
  year = {2024},
  doi = {10.1007/s10462-024-10916-x},
}
Figure