Abstract
Recent advances in deep neural networks have achieved outstanding success in natural language processing. Due to the success and the black-box nature of the deep models, interpretation methods that provide insight into the decision-making process of the models have received an influx of research attention. However, there is no quantitative evaluation comparing interpretation methods for text classification other than observing classification accuracy or prediction confidence when important word grams are removed. This is due to the lack of interpretation ground truth. Manual labeling of a large interpretation ground truth is time-consuming. We propose IDC, a new benchmark for quantitative evaluation of Interpretation methods for Deep text Classification models. IDC consists of three methods that take existing text classification ground truth and generate three corresponding pseudo-interpretation ground truth (PGT) datasets. We propose to use interpretation recall, interpretation precision, and Cohen’s kappa inter-agreement as performance metrics. We used the pseudo ground truth datasets and the metrics to evaluate six interpretation methods.
Introduction
Deep neural networks have revolutionized natural language processing (NLP), yet they remain largely opaque "black boxes." To address this, researchers have developed local interpretation methods designed to highlight the specific words or phrases that drive a model's prediction. However, evaluating these interpretation methods themselves is a significant challenge. Without a definitive "ground truth" of which words are truly important, researchers have relied on proxy metrics, such as measuring the drop in model confidence when highlighted words are removed. This paper introduces IDC, a benchmark designed to provide a scalable, quantitative framework for evaluating text interpretation.
The Challenge of Evaluation
The primary obstacle in this field is the lack of interpretation ground truth. Human annotation is not only slow and expensive but also surprisingly inconsistent. In this study, the authors found that two human experts achieved a Cohen’s kappa agreement of only 0.39 when identifying important words in the same dataset. This subjectivity makes human labeling a flawed gold standard. The IDC benchmark addresses this by automatically generating Pseudo-interpretation Ground Truth (PGT) from existing datasets.
The IDC Framework: Generating PGTs
IDC utilizes three distinct methods to generate automated baselines for word importance:
1. Weighted Gram Activation (WGA)
This method employs a Convolutional Neural Network (CNN) and a gradient-by-input approach. By analyzing the gradients of convolutional filters, WGA identifies word-grams that are highly relevant to a document's correct classification.
As shown in the figure above, the process involves mapping gradients ($GM^c$) and feature maps ($FM$) to generate a final heatmap that highlights the most influential tokens in the input text.
2. Mixed Integer Linear Program (MILP)
Unlike WGA, MILP does not rely on a trained classifier. Instead, it treats word importance as an optimization problem. Using TF-IDF scores, it mathematically identifies the smallest set of words that appear consistently in one class but rarely in others.
3. Hybrid Method
The Hybrid approach combines the strengths of both previous methods. It uses the relevance scores from WGA as inputs for the MILP optimizer, refining the selection of class-specific words.
Experimental Results
The authors evaluated six state-of-the-art interpretation methods—Saliency Map, Grad-CAM, Integrated Gradient, DeepLIFT, Layer-wise Relevance Propagation (LRP), and Hierarchical Attention—against the generated PGTs using the Movie Review (MR) dataset.
- Top Performer: Layer-wise Relevance Propagation (LRP) emerged as the most effective method, achieving a Cohen’s kappa of 0.59 and a precision of 0.72 when compared against the WGA-PGT.
- The Attention Paradox: Interestingly, Hierarchical Attention performed the worst among the tested methods. This provides quantitative evidence for the ongoing debate in the NLP community regarding whether attention weights truly represent feature importance.
- Validation: To confirm the validity of the PGTs, the authors removed the identified words from the test set. This caused the model's relative confidence to drop significantly (from 0.75 to 0.52), proving that the PGTs successfully captured the words driving the model's logic.
Conclusion
The IDC benchmark provides a much-needed quantitative foundation for the field of explainable AI in NLP. By replacing subjective human labeling with automated, mathematically grounded pseudo-ground truths, IDC allows for a more rigorous comparison of interpretation methods. While computational constraints currently limit the scale of the MILP and Hybrid methods, future work with parallelized solvers promises to further enhance the quality and scale of these evaluation datasets.
Summary updated