TL;DR
As Natural Language Processing (NLP) models become increasingly complex and powerful, understanding why they make certain predictions is crucial for trust and development. This survey provides a comprehensive overview of the current landscape of Explainable AI (XAI) specifically within NLP, categorizing existing approaches and highlighting key techniques, operations, and visualization methods. It serves as a valuable resource for researchers and developers looking to build more interpretable NLP systems.
What Problem Does This Paper Address?
Modern NLP models, particularly deep learning architectures, have achieved remarkable performance gains. However, this advancement has often come at the cost of interpretability, turning these models into "black boxes." This lack of transparency can erode user trust, especially in applications like chatbots, recommendation systems, and information retrieval. While Explainable AI (XAI) is a growing field, there's a need for a focused survey on its application within NLP, given the unique challenges and approaches in this domain. This paper aims to fill that gap by surveying recent NLP XAI research, providing a structured understanding of the field, and identifying areas for future development.
Methods
The authors conducted a systematic survey of NLP papers focusing on explainability, primarily from major NLP conferences between 2013 and 2019. They filtered papers based on keywords like "explainability," "interpretability," and "transparent." After an initial review, 50 papers were selected for in-depth analysis.
The survey categorizes explanations along two main axes:
- Local vs. Global: Whether the explanation is for a single prediction (local) or the model's overall behavior (global).
- Self-Explaining vs. Post-Hoc: Whether the explanation is generated directly by the model during prediction (self-explaining) or requires a separate process after prediction (post-hoc).
They also analyze:
- Explainability Techniques: Methods used to generate explanations, including Feature Importance, Surrogate Models, Example-Driven approaches, Provenance-Based methods, and Declarative Induction.
- Operations to Enable Explainability: Underlying mechanisms like First-Derivative Saliency, Layer-wise Relevance Propagation, Input Perturbations, and Attention mechanisms.
- Visualization Techniques: How explanations are presented to users, such as Saliency (heatmaps, highlighting), Raw Declarative Representations (rules, programs), and Natural Language Explanations.
Datasets
The paper does not focus on specific datasets for training or evaluation but rather on the techniques and approaches used in existing NLP research. The survey analyzes papers that have applied these XAI methods to various NLP tasks, including text classification, question answering, and sentiment analysis. No new benchmarks or datasets were introduced in this survey.
Key Insights and Ideas
- The "Black Box" Problem is Pervasive: The increasing complexity of NLP models has made them less interpretable, creating a critical need for XAI.
- NLP-Specific XAI is Emerging: While XAI is a broad field, this survey highlights the unique challenges and solutions within NLP, moving beyond general XAI surveys.
- Local Explanations Dominate: The vast majority of surveyed papers focus on explaining individual predictions (local explanations) rather than the model's overall behavior (global).
- Feature Importance is King (for now): Feature importance-based techniques, often coupled with saliency visualizations, are the most frequently used methods. This is attributed to features being fundamental building blocks and the widespread use of attention mechanisms in NLP.
- Visualization Matters: How explanations are presented is crucial for user understanding. Saliency-based visualizations are popular for their visual perceptibility, while natural language explanations aim for broader accessibility.
- Evaluation is Nascent: The field is still grappling with standardized methods for evaluating explanation quality, with many papers relying on informal examination or lacking formal evaluation altogether.
Key Results
- Out of 107 initially identified papers, 50 were selected for detailed analysis, focusing on XAI in NLP.
- Local Explanations: 46 out of 50 papers focused on local explanations.
- Global Explanations: Only 4 papers focused on global explanations.
- Feature Importance: This technique was used in 29 papers, making it the most common.
- Surrogate Models: Used in 8 papers.
- Saliency: The most dominant visualization technique, used in a majority of papers.
- Evaluation: 32 papers used informal examination, 12 compared to ground truth, and 9 used human evaluation.
Limitations and Future Work
- Lack of Formal Definitions and Evaluation: A significant gap identified is the absence of standardized definitions and robust evaluation metrics for explainability in NLP. This makes it difficult to compare approaches objectively.
- Terminology Clarity: The survey calls for clearer terminology and a better understanding of what constitutes explainability and how it relates to different target audiences.
- Fidelity and Causality: More critical examination is needed to ensure that claimed explanations truly reflect the model's prediction process (fidelity) and to explore causal relationships.
- Underrepresentation of Global Explanations: The survey found a surprisingly low number of papers focusing on global explanations, despite the historical importance of interpretable white-box models in NLP. The authors suggest this might be an oversight, as white-box models could serve as valuable testbeds for evaluating explanation techniques.
- Trade-offs between Explainability and Performance: Further research is needed to understand the potential trade-offs between model performance and explainability, as some studies suggest a loss in predictive quality with increased interpretability, while others claim the opposite.
Summary updated