Abstract: Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing (NLP). We discuss the main categorization of explanations, as well as the various ways explanations can be arrived at and visualized. We detail the operations and explainability techniques currently available for generating explanations for NLP model predictions, to serve as a resource for model developers in the community. Finally, we point out the current gaps and encourage directions for future work in this important research area.
Introduction
Modern Natural Language Processing (NLP) has undergone a paradigm shift from transparent "white-box" systems, such as rule-based engines, to high-performance but opaque "black-box" deep learning architectures. While these models achieve state-of-the-art results, their lack of transparency erodes user trust and complicates debugging for developers. This survey from IBM Research addresses the critical need for understanding why a model makes specific predictions, establishing a unified taxonomy for XAI in the NLP domain.
A Taxonomy for XAI in NLP
The authors propose a multi-dimensional taxonomy to categorize XAI research, focusing primarily on the "outcome explanation problem"—justifying specific model outputs rather than the entire model's logic.
Categorization
Explanations are categorized along two primary axes: scope and timing.
- Local vs. Global: Local explanations justify a single prediction, while global explanations attempt to describe the entire model's behavior. The survey finds that the vast majority of current NLP work (46 out of 50 papers) is local.
- Self-Explaining vs. Post-Hoc: Self-explaining models incorporate interpretability into their architecture (e.g., Attention mechanisms), whereas post-hoc methods generate explanations through a separate process after a prediction is made.
Derivation Techniques and Operations
The "raw" mathematical explanation can be generated through several techniques:
- Feature Importance: Identifying the most influential tokens (e.g., via gradients or attention).
- Surrogate Models: Using simpler, interpretable models to approximate complex ones.
- Example-driven: Citing similar labeled examples from training data.
- Provenance-based: Showing step-by-step reasoning, often seen in mathematical QA.
- Declarative Induction: Extracting human-readable rules or programs.
These techniques rely on underlying operations such as First-derivative saliency (gradients), Attention weights, and Input perturbations (observing output changes after modifying inputs).
Key Insights and Results
The survey analyzed 50 papers from major NLP conferences (2013-2019), revealing several critical trends:
- Dominance of Feature Importance: This is the most popular technique (29/50 papers), typically visualized through saliency heatmaps or highlighting.
- The Attention Controversy: While widely used, there is significant debate over whether "Attention is Explanation," as attention weights do not always align with the features actually driving the model's decision.
- The Evaluation Crisis: A striking "evaluation gap" exists; over 60% of research relies on informal "gut checks" by the authors. Only a small fraction utilizes human evaluation or ground-truth explanation datasets.
Future Directions and Limitations
The authors identify three major gaps that the community must address:
- Standardized Terminology: A lack of formal definitions for "good" explanations hinders progress.
- Rigorous Benchmarking: The field needs metrics that move beyond subjective examination to ensure objective evaluation.
- Fidelity vs. Plausibility: There is a risk of generating "plausible" explanations that humans find convincing but which do not faithfully represent the model's internal causal process. Future research must prioritize fidelity to ensure explanations are truly representative of model logic.
Summary updated