Pathologies of Neural Models Make Interpretations Difficult

Shi Feng, Eric Wallace, Alvin Grissom Ii, Mohit Iyyer, Pedro Rodriguez, Jordan Boyd-GraberEMNLP 2018

TL;DR

This paper introduces "input reduction," a novel method to probe neural network interpretability by iteratively removing less important words from an input. They discover that models remain highly confident in their predictions even when the input becomes nonsensical to humans, revealing a significant disconnect between model behavior and human understanding. The authors propose entropy regularization on these reduced inputs as a way to mitigate these pathologies and improve interpretability without sacrificing accuracy.

What Problem Does This Paper Address?

Interpreting neural networks, especially in Natural Language Processing (NLP), is crucial for understanding their decision-making processes and building trust. A common approach is "feature attribution," where methods highlight the most important input features (e.g., words) that contribute to a model's prediction. These methods often rely on techniques like input perturbation (removing a word and observing the confidence drop) or gradients. However, the authors argue that these methods might be overlooking critical issues. They question whether the words deemed "important" by these methods truly align with human intuition and whether the model's confidence is a reliable indicator of its understanding, especially when inputs are degraded. They aim to expose these limitations and propose a way to improve interpretability by addressing the underlying pathologies.

Methods

The core contribution is input reduction. This process iteratively removes words from an input sentence, starting with the least important ones, while ensuring the model's prediction remains unchanged. Importance is measured using gradient-based methods, specifically the dot product of a word's embedding and the gradient of the output with respect to that embedding. This is an efficient approximation of more computationally expensive methods like leave-one-out.

To make the reduction process more effective and find shorter, potentially more nonsensical inputs, they augment input reduction with beam search. This allows for the removal of a small set of the least important words at each step, rather than just one.

To address the observed pathologies, they propose entropy regularization during fine-tuning. This objective encourages the model to have higher entropy (more uncertainty) on the reduced, nonsensical examples, aiming to make its confidence a more reliable indicator of its understanding.

Datasets

The paper evaluates their methods on three standard NLP datasets:

  • SQuAD (Stanford Question Answering Dataset): For reading comprehension.
  • SNLI (Stanford Natural Language Inference): For textual entailment.
  • VQA (Visual Question Answering): For image-based question answering.

They use the validation sets of these datasets for their experiments. No new datasets are introduced.

Key Insights and Ideas

  • Models are "confidently wrong" on nonsensical inputs: The most striking finding is that neural models can maintain high confidence in their predictions even when the input is reduced to a few words that are completely nonsensical to humans. This suggests that models might be learning spurious correlations or relying on superficial cues rather than genuine understanding.
  • Input reduction exposes "rubbish examples": The reduced inputs generated by the method fit the definition of "rubbish examples" – inputs that humans find impossible to classify but for which models produce confident predictions. This highlights a failure in the model's uncertainty estimation.
  • Interpretability methods can be misled by overconfidence: Standard feature attribution methods, which rely on model confidence, can be fooled by these pathologies. The words they highlight as important might not be the ones that truly enable human understanding, especially when the input is degraded.
  • Second-order sensitivity matters: The paper points out that while gradient-based methods capture first-order sensitivity (how predictions change with small input changes), they might miss crucial second-order effects (how interpretations change). Input reduction, by iteratively perturbing the input, can reveal these deeper issues.
  • Entropy regularization can improve interpretability: By encouraging models to be less confident on nonsensical inputs, entropy regularization can make the reduced examples more reasonable to humans and improve the alignment between model and human interpretations.

Key Results

  • Input reduction consistently shortens inputs (often to one or two words) across SQuAD, SNLI, and VQA without significantly decreasing model confidence.
  • Human experiments show a dramatic drop in accuracy on these reduced inputs compared to original ones, indicating humans cannot make reliable predictions.
  • The reduced examples are preferred by humans over randomly generated short inputs after fine-tuning with entropy regularization.
  • Fine-tuning with entropy regularization on reduced examples improves human accuracy on these examples and makes them appear more reasonable, without sacrificing model accuracy on regular examples. For instance, human accuracy on reduced SQuAD examples increased from 31.72% to 51.61% after fine-tuning.

Limitations and Future Work

  • Generalizability of findings: While the pathologies are observed across multiple tasks and architectures, the authors acknowledge that further work is needed to explore their prevalence in other domains and model types.
  • Necessity of input reduction: The paper provides a baseline comparison, but it's not definitively proven that input reduction is the only way to generate these negative examples for regularization.
  • Future directions: The authors suggest several avenues for future research:
  • More thorough evaluation of interpretation methods.
  • Developing better uncertainty and confidence estimation techniques.
  • Moving beyond simple bag-of-word heatmaps for interpretation.
  • Investigating new training objectives that explicitly incorporate uncertainty estimation.

Summary updated

BibTeX
@inproceedings{feng2018pathologies,
  author = {Shi Feng and Eric Wallace and Alvin Grissom II and Mohit Iyyer and Pedro Rodriguez and Jordan Boyd-Graber},
  title = {{Pathologies of Neural Models Make Interpretations Difficult}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure