Using model explanations to guide deep learning models towards consistent explanations for EHR data

Matthew Watson, Bashar Hasan, Noura Al MoubayedScientific 2022 · DOI

TL;DR

Deep learning models, especially in critical fields like healthcare, often produce inconsistent explanations even with identical architectures, hindering trust and adoption. This paper introduces a novel "explanation ensemble" architecture that trains sub-models to generate consistent explanations, significantly improving reliability without sacrificing performance.

What Problem Does This Paper Address?

In fields like healthcare and finance, where decisions have high stakes, understanding why a deep learning model makes a prediction is as crucial as the prediction itself. This is where "explainability" comes in. However, a significant challenge is that even identical deep learning models, when trained with minor variations (like different random seeds or data shuffling), can produce vastly different explanations. This inconsistency erodes trust, making it difficult for domain experts and end-users to rely on these models. The paper tackles this "explanation inconsistency" problem, particularly for tabular data commonly found in Electronic Health Records (EHRs).

Methods

The core innovation is a novel deep learning ensemble architecture called the "explanation ensemble." Instead of training a single model, this approach trains multiple sub-models (e.g., 10) with the same architecture but different hyperparameters (specifically, random seeds or training set order). The key is a specially designed training process that uses a discriminator. This discriminator's job is to learn to distinguish which sub-model generated a given explanation. The sub-models, in turn, are trained to "fool" this discriminator by producing explanations that are similar to each other. This forces them to rely on features that are consistently important across all sub-models, effectively averaging out spurious correlations and leading to more stable explanations. The final model's prediction is an average of the sub-models' outputs.

Datasets

The study evaluates the proposed method on six tabular datasets across four tasks:

  • Breast Cancer Wisconsin (BCW): Binary classification.
  • KAIMRC: Binary classification and regression tasks related to diabetes prediction.
  • MIMIC-IV: Binary classification for mortality prediction (a time-series dataset).
  • Codon Usage (Kingdom & DNA): Multi-class classification tasks related to genomic data.

These datasets are chosen for their relevance to healthcare and their tabular nature, which simplifies explanation analysis.

Key Insights and Ideas

  • Stochasticity is the Root Cause: The paper strongly suggests that the inherent randomness in the training process (e.g., random seed, data shuffling) is the primary driver of explanation inconsistency.
  • Ensembling for Consistency: While standard ensembling improves model robustness, it doesn't inherently fix explanation inconsistency. The proposed "explanation ensemble" specifically targets this by making sub-models learn consistent explanations.
  • Discriminator as a Consistency Driver: The discriminator, trained to identify the origin of explanations, acts as a powerful signal to guide sub-models towards shared feature importance, even when trained with different initial conditions.
  • Trade-off is Worth It: While explanation ensembles might introduce a slight decrease in raw performance (within 10%), the significant gains in explanation consistency are argued to be a worthwhile trade-off for applications demanding high trust.

Key Results

The explanation ensemble architecture demonstrated substantial improvements in explanation consistency:

  • MIMIC-IV: Consistency improved by as much as 315% (from 0.02433 to 0.1011).
  • BCW: Consistency improved by 124% (from 0.12282 to 0.4450).
  • Across all tested tasks, the explanation ensemble consistently outperformed baseline models and standard ensembles in terms of explanation consistency, as measured by both a custom metric (C) and Jensen-Shannon Divergence (CJSD).
  • Statistical significance tests (Mann-Whitney U) confirmed that the improvements in explanation consistency were statistically significant across all datasets.
  • Model performance remained largely comparable, with only a slight, acceptable decrease in some cases.

Limitations and Future Work

  • Tabular Data Focus: The current study is limited to tabular datasets. The authors suggest future work should explore the applicability of their techniques to other data modalities like images and text, and to more complex model architectures.
  • Dataset Properties and Causality: The paper raises an interesting question about what properties of a dataset make it easier to learn consistent (and potentially causal) features. Future research could delve into this relationship and investigate the degree to which their method promotes causal feature learning over mere correlation.
  • Computational Cost: While the proposed method significantly reduces inference time costs through averaging techniques (like checkpoint averaging), the initial training of multiple sub-models is still more computationally intensive than training a single model.
  • Further Consistency Improvements: While significant improvements were observed, consistency for some datasets remains relatively low, indicating room for further methodological advancements.

Summary updated

Abstract

It has been shown that identical deep learning (DL) architectures will produce distinct explanations when trained with different hyperparameters that are orthogonal to the task (e.g. random seed, training set order). In domains such as healthcare and finance, where transparency and explainability is paramount, this can be a significant barrier to DL adoption. In this study we present a further analysis of explanation (in)consistency on 6 tabular datasets/tasks, with a focus on Electronic Health Records data. We propose a novel deep learning ensemble architecture that trains its sub-models to produce consistent explanations, improving explanation consistency by as much as 315% (e.g. from 0.02433 to 0.1011 on MIMIC-IV), and on average by 124% (e.g. from 0.12282 to 0.4450 on the BCW dataset). We evaluate the effectiveness of our proposed technique and discuss the implications our results have for both industrial applications of DL and explainability as well as future methodological work.
BibTeX
@inproceedings{watson2022using,
  author = {Matthew Watson and Bashar Hasan and Noura Al Moubayed},
  title = {{Using model explanations to guide deep learning models towards consistent explanations for EHR data}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-24356-6},
}
Figure