Interpreting Text Classifiers by Learning Context-sensitive Influence of Words

Sawan Kumar, Kalpit Dixit, Kashif ShahNAACL 2021

TL;DR

This paper introduces MOXIE, a novel framework for interpreting text classifiers by learning the context-sensitive influence of words. MOXIE goes beyond traditional importance scores to provide testable predictions for counterfactuals and biases, offering a more robust and interactive way to understand and trust AI models.

What Problem Does This Paper Address?

Interpreting complex text classification models, especially large neural networks, is crucial for building trust and understanding their behavior. Current methods often provide importance scores for words, but these scores are difficult to test or use to improve the interpretation method itself, leading to a compounding problem of opacity. Furthermore, importance scores on individual examples are insufficient for a comprehensive understanding of model behavior. There's a need for scalable, testable methods that can reveal model biases and allow for deeper interaction beyond just assigning scores.

Methods

MOXIE employs a "student-teacher" model architecture. The "teacher" is the text classifier being interpreted, and the "student" is a new model trained to approximate the teacher's behavior. The core idea is to learn the difference between the teacher's representation of an input text and its representation when a specific word is masked. This difference serves as a proxy for the context-sensitive influence of that word.

Key components of MOXIE include:

  • Context Processor (AC): Processes the input text with a masked word.
  • Token Processor (AT): Processes the masked word itself.
  • Combine Module (AM): Integrates the outputs of AC and AT to predict the influence.

MOXIE is trained with a global learning objective, minimizing the mean squared error between the teacher's original representation and the student's predicted representation. This global objective allows for straightforward testing and improvement of the interpretation method itself using standard ML pipelines.

MOXIE enables answering three types of questions:

  1. Importance Scores: Quantifying the influence of words on a prediction.
  2. Counterfactuals: Identifying minimal changes to input text that alter the model's decision.
  3. Biases: Detecting systematic biases against certain words or phrases.

Datasets

The primary dataset used for training and evaluation is the Stanford Sentiment Treebank-2 (SST-2), a benchmark for binary sentiment classification of movie reviews.

Key Insights and Ideas

  • Beyond Importance Scores: The paper argues that importance scores alone are insufficient and difficult to evaluate. MOXIE's strength lies in generating testable predictions for counterfactuals and biases, which can be directly verified by the model being interpreted.
  • Global Learning Objective: Training the interpretation model with a global objective (minimizing representation difference) provides a clear path for testing and improving the interpretation method itself, unlike heuristic-based approaches.
  • Context-Sensitive Influence: MOXIE explicitly models how a word's influence changes based on its surrounding context, which is crucial for nuanced understanding.
  • Efficiency for Bias Detection: MOXIE significantly speeds up bias detection by pre-computing representations, enabling interactive exploration of biases against specific word groups (e.g., LGBTQ+ terms).

Key Results

  • Student Model Accuracy: The MOXIE student model achieved significant gains in "all-correct accuracy" over a context-only baseline, demonstrating its ability to learn context-token interactions. For instance, with RoBERTa-base as the teacher, the student model achieved 89.24% accuracy compared to a baseline of 78.64%.
  • Importance Scores: MOXIE's influence scores and unlike ratios were found to be competitive with LIME, a popular method, even though MOXIE did not see test examples or their neighborhoods during training.
  • Counterfactual Reliability: MOXIE demonstrated reliable counterfactual prediction, with a median accuracy of 90% across contexts where at least one counterfactual was found.
  • Bias Detection: MOXIE successfully identified potential biases against LGBTQ+ words (e.g., "lesbian" compared to "straight"), showing a significantly larger set of contexts where the probe word led to a negative prediction. The student model achieved up to 90.0% accuracy in identifying these biased contexts.
  • Speedup: MOXIE offers a substantial speedup (~450x) for bias probing compared to using the teacher model directly, enabling interactive analysis.

Limitations and Future Work

  • Student Model Accuracy: The authors acknowledge that improving the accuracy of the student model is a key area for future work, as higher accuracy would lead to more reliable interpretations.
  • Counterfactual Analysis: Further analysis of the nature of counterfactuals selected by the student model could yield insights for improving the interpretation method.
  • New Learning Objectives: Exploring other learning objectives that enable testable predictions is identified as a challenging but useful future direction.
  • Exhaustiveness of Bias Detection: The bias detection study for LGBTQ+ words is noted as a representative sample, not an exhaustive list. The word "queer" showed no predicted bias, possibly due to its additional meanings.

Summary updated

BibTeX
@inproceedings{kumar2021interpreting,
  author = {Sawan Kumar and Kalpit Dixit and Kashif Shah},
  title = {{Interpreting Text Classifiers by Learning Context-sensitive Influence of Words}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure