TL;DR
Deep learning models, especially in critical fields like healthcare, often produce inconsistent explanations even with identical architectures, hindering trust and adoption. This paper introduces a novel "explanation ensemble" architecture that trains sub-models to generate consistent explanations, significantly improving reliability without sacrificing performance.
What Problem Does This Paper Address?
In fields like healthcare and finance, where decisions have high stakes, understanding why a deep learning model makes a prediction is as crucial as the prediction itself. This is where "explainability" comes in. However, a significant challenge is that even identical deep learning models, when trained with minor variations (like different random seeds or data shuffling), can produce vastly different explanations. This inconsistency erodes trust, making it difficult for domain experts and end-users to rely on these models. The paper tackles this "explanation inconsistency" problem, particularly for tabular data commonly found in Electronic Health Records (EHRs).
Methods
The core innovation is a novel deep learning ensemble architecture called the "explanation ensemble." Instead of training a single model, this approach trains multiple sub-models (e.g., 10) with the same architecture but different hyperparameters (specifically, random seeds or training set order). The key is a specially designed training process that uses a discriminator. This discriminator's job is to learn to distinguish which sub-model generated a given explanation. The sub-models, in turn, are trained to "fool" this discriminator by producing explanations that are similar to each other. This forces them to rely on features that are consistently important across all sub-models, effectively averaging out spurious correlations and leading to more stable explanations. The final model's prediction is an average of the sub-models' outputs.
Datasets
The study evaluates the proposed method on six tabular datasets across four tasks:
- Breast Cancer Wisconsin (BCW): Binary classification.
- KAIMRC: Binary classification and regression tasks related to diabetes prediction.
- MIMIC-IV: Binary classification for mortality prediction (a time-series dataset).
- Codon Usage (Kingdom & DNA): Multi-class classification tasks related to genomic data.
These datasets are chosen for their relevance to healthcare and their tabular nature, which simplifies explanation analysis.
Key Insights and Ideas
- Stochasticity is the Root Cause: The paper strongly suggests that the inherent randomness in the training process (e.g., random seed, data shuffling) is the primary driver of explanation inconsistency.
- Ensembling for Consistency: While standard ensembling improves model robustness, it doesn't inherently fix explanation inconsistency. The proposed "explanation ensemble" specifically targets this by making sub-models learn consistent explanations.
- Discriminator as a Consistency Driver: The discriminator, trained to identify the origin of explanations, acts as a powerful signal to guide sub-models towards shared feature importance, even when trained with different initial conditions.
- Trade-off is Worth It: While explanation ensembles might introduce a slight decrease in raw performance (within 10%), the significant gains in explanation consistency are argued to be a worthwhile trade-off for applications demanding high trust.
Key Results
The explanation ensemble architecture demonstrated substantial improvements in explanation consistency:
- MIMIC-IV: Consistency improved by as much as 315% (from 0.02433 to 0.1011).
- BCW: Consistency improved by 124% (from 0.12282 to 0.4450).
- Across all tested tasks, the explanation ensemble consistently outperformed baseline models and standard ensembles in terms of explanation consistency, as measured by both a custom metric (C) and Jensen-Shannon Divergence (CJSD).
- Statistical significance tests (Mann-Whitney U) confirmed that the improvements in explanation consistency were statistically significant across all datasets.
- Model performance remained largely comparable, with only a slight, acceptable decrease in some cases.
Limitations and Future Work
- Tabular Data Focus: The current study is limited to tabular datasets. The authors suggest future work should explore the applicability of their techniques to other data modalities like images and text, and to more complex model architectures.
- Dataset Properties and Causality: The paper raises an interesting question about what properties of a dataset make it easier to learn consistent (and potentially causal) features. Future research could delve into this relationship and investigate the degree to which their method promotes causal feature learning over mere correlation.
- Computational Cost: While the proposed method significantly reduces inference time costs through averaging techniques (like checkpoint averaging), the initial training of multiple sub-models is still more computationally intensive than training a single model.
- Further Consistency Improvements: While significant improvements were observed, consistency for some datasets remains relatively low, indicating room for further methodological advancements.
Summary updated