Contextual explanation rules for neural clinical classifiers

Madhumita Sushil, Simon Suster, Walter DaelemansNAACL 2021

TL;DR

This paper introduces UNRAVEL, a novel pipeline for generating interpretable "decision list" explanations for recurrent neural networks (RNNs), particularly those using word embeddings. UNRAVEL overcomes limitations of prior methods by incorporating word order and contextual information through "skipgrams," leading to more faithful and understandable explanations for clinical and sentiment analysis tasks.

What Problem Does This Paper Address?

Understanding the decisions of complex neural networks, especially in critical domains like healthcare, is crucial for building trust, improving models, and ensuring safety. While many interpretability techniques highlight important input features, they often fail to capture how these features interact. Rule-based explanations offer a way to represent these interactions, but existing methods struggle with the sequential nature and word embeddings common in modern NLP models. Specifically, previous approaches often treat models as black boxes, fitting separate interpretable models without leveraging the original model's parameters, which can lead to expensive global explanations and a disconnect between the explained and explaining models. Furthermore, capturing the nuances of word order and context in explanations for sequential data like text has been a significant challenge.

Methods

The core of this paper is the UNRAVEL pipeline, which aims to generate interpretable explanation rules for RNNs. The process involves several key steps:

  1. Word Importance Computation: Instead of relying on raw gradients for each embedding dimension (which are hard to interpret), UNRAVEL pools these gradients to derive a single importance score for each word. They explore L2 norm, sum, and dot product pooling, ultimately favoring the dot product for its ability to account for embedding values and produce more intuitive importance distributions.
  2. Skipgrams for Context: To incorporate word order and context, UNRAVEL represents input documents using "skipgrams" – contiguous subsequences of words of varying lengths (1-4 words). The importance of a skipgram is derived from the importance of its constituent words. This allows explanations to capture phrases and their relationships, rather than just individual words.
  3. Discretization and Rule Induction: The importance scores of the top skipgrams are then discretized into five levels (e.g., ++, +, 0, -, --) to represent their impact on the predicted output. These discretized skipgrams, along with the model's predictions, are fed into a rule induction module (specifically, PART, a simplified decision tree learner) to generate decision lists. These lists are designed to be understandable independently of rule order.
  4. Transferable Explanations: A crucial aspect is ensuring explanations are transferable. UNRAVEL trains the explanation rules solely on the training data, hypothesizing that these rules should generalize to unseen data without needing prior knowledge of the test set.

Datasets

The authors introduce a synthetic sepsis-identification dataset to address the challenge of evaluating explanation methods. This dataset is constructed based on existing medical knowledge and clinical corpora, simulating real-world complexity while allowing for controlled evaluation. They also apply their method to two real-world clinical datasets for sepsis classification and a sentiment analysis dataset (Stanford Sentiment Treebank - SST2).

Key Insights and Ideas

  • Skipgrams are Key for Context: Moving beyond individual words to skipgrams significantly improves the interpretability of explanations by capturing word order and contextual relationships, which are vital for understanding sequential data.
  • Gradient-Informed Rules are More Faithful: Leveraging model gradients (via dot product pooling) for importance scores leads to explanations that are more faithful to the model's predictions compared to baseline methods that ignore model parameters.
  • Transferable Explanations are Achievable: The hypothesis that explanations can be learned from training data alone and generalize to unseen data is supported, offering a more robust approach to interpretability.
  • Synthetic Data for Rigorous Evaluation: The creation of a synthetic dataset is a valuable contribution, enabling quantitative and qualitative evaluation of explanation techniques in a controlled environment.

Key Results

  • High Fidelity: UNRAVEL consistently achieves high explanation fidelity scores (macro F1) across various LSTM classifiers and datasets, often significantly outperforming baseline methods. For instance, on the synthetic dataset, UNRAVEL achieved fidelity scores around 99%, while a baseline approach reached only ~83%.
  • Reduced Complexity: UNRAVEL generates explanations with substantially fewer rules compared to baseline methods, making them more concise and easier to comprehend. For example, UNRAVEL produced around 13-51 rules, whereas baselines generated 60-63 rules.
  • Clinical Relevance: The pipeline successfully extracts meaningful explanations for clinical tasks, identifying key terms and their interactions that align with medical knowledge. For sepsis classification, explanations correctly identified the importance of infection and inflammatory response terms.
  • Sentiment Analysis Performance: On the SST2 dataset, UNRAVEL provided more accurate explanations than the Anchors method.

Limitations and Future Work

  • Skipgram Length and Complexity: The choice of skipgram length (1-4) and the number of top skipgrams retained (5k) were manually decided to balance context and complexity. Increasing these values could lead to sparser phrases and a larger explanation vocabulary, potentially impacting computational complexity and explanation coverage.
  • Scalability to More Complex Models: While UNRAVEL works well for RNNs, the authors suggest future work could extend its capabilities to classifiers with even more complex patterns.
  • Automated Evaluation Challenges: While the synthetic dataset aids evaluation, human evaluation remains important, and the authors note that human evaluation is not always ideal or feasible in complex domains.
  • Future Directions: The authors propose extending the approach to achieve more accurate, less complex, and more scalable explanations for models with more intricate patterns.

Summary updated

BibTeX
@inproceedings{sushil2021contextual,
  author = {Madhumita Sushil and Simon Suster and Walter Daelemans},
  title = {{Contextual explanation rules for neural clinical classifiers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure