Abstract
We study the invariance characteristics of pre-trained predictive models by empirically learning transformations on the input that leave the prediction function approximately unchanged. To learn invariant transformations, we minimize the Wasserstein distance between the predictive distribution conditioned on the data instances and the predictive distribution conditioned on the transformed data instances. To avoid finding degenerate or perturbative transformations, we add a similarity regularization to discourage similarity between the data and its transformed values. We theoretically analyze the correctness of the algorithm and the structure of the solutions. Applying the proposed technique to clinical time series data, we discover variables that commonly-used LSTM models do not rely on for their prediction, especially when the LSTM is trained to be adversarially robust. We also analyze the invariances of BioBERT on clinical notes and discover words that it is invariant to.
Introduction
Deep learning models are increasingly deployed in high-stakes healthcare environments for tasks such as mortality prediction and length-of-stay estimation. Despite their performance, these models often function as "black boxes," making it difficult for clinicians to understand the underlying logic of their predictions. While traditional attribution methods highlight which features a model uses for specific instances, they often fail to provide a global perspective on what a model has learned to ignore. Understanding these global invariances is critical for ensuring model safety, debugging potential biases, and identifying reliance on "shortcuts" like missing data indicators.
The MIND Framework
To address this, we introduce the Model INvariance Discovery (MIND) framework. MIND identifies what a model disregards by learning transformations ($T_\phi$) that alter input data ($X$) without significantly changing the model's output ($p_\theta$).
The framework optimizes a dual-objective function:
- Prediction Preservation: It minimizes the Wasserstein distance between the model’s predictions on original and transformed data.
- Similarity Regularization: To prevent the algorithm from simply learning an identity transformation (doing nothing), a regularization term forces the transformed data to be as different as possible from the original input.
MIND employs two primary transformation types: Gating Transformations, which assign a "MIND Score" (0 to 1) to features to indicate sensitivity, and Residual Transformations, which utilize neural networks to identify complex, non-linear invariances.
Clinical Insights and Invariances
Applying MIND to the MIMIC-III dataset for in-hospital mortality prediction revealed significant insights into model behavior. The framework discovered that standard LSTM models were highly invariant to features like "height" and "capillary refill rate." While the invariance to height is expected in an ICU setting, the invariance to capillary refill was identified as a potential blind spot caused by extremely sparse data (recorded in only 0.28% of cases).
Conversely, features such as the Glasgow Coma Scale (GCS) and $SpO_2$ exhibited high MIND scores, indicating that the model is highly sensitive to these physiological signals. Interestingly, the framework also highlighted a "missingness shortcut": models often rely more on the presence of a measurement (the missing value indicator) than the actual value itself, particularly for features like $SpO_2$.
Robustness and Temporal Patterns
Our analysis showed that adversarial training—training a model to resist small perturbations—actually increases its invariance to noise and missing value indicators. This shift forces the model to rely more heavily on actual physiological signals, improving its reliability. Furthermore, we found that LSTMs exhibit a strong "recency bias," prioritizing data from the final quarter of a patient's stay, whereas Transformer-based models distribute their attention more evenly across the entire time series.
In the realm of clinical notes, applying MIND to BioBERT revealed that the model is most sensitive to medically significant tokens (e.g., "pneumonia," "infant") while remaining invariant to neutral or procedural language (e.g., "compared," "please").
Conclusion
The MIND framework provides a theoretically grounded and computationally efficient method for auditing black-box models in healthcare. By identifying what models ignore, MIND serves as a powerful debugging tool, helping researchers detect spurious correlations and move toward more robust, clinically grounded predictive systems.
Summary updated