Understanding Behavior of Clinical Models under Domain Shifts

Jayaraman J. Thiagarajan, Deepta Rajan, Prasanna Sattigeri

Abstract

The hypothesis that computational models can be reliable enough to be adopted in prognosis and patient care is revolutionizing healthcare. Deep learning, in particular, has been a game changer in building predictive models, thus leading to community-wide data curation efforts. However, due to inherent variabilities in population characteristics and biological systems, these models are often biased to the training datasets. This can be limiting when models are deployed in new environments, when there are systematic domain shifts not known a priori. In this paper, we propose to emulate a large class of domain shifts, that can occur in clinical settings, with a given dataset, and argue that evaluating the behavior of predictive models in light of those shifts is an effective way to quantify their reliability. More specifically, we develop an approach for building realistic scenarios, based on analysis of disease landscapes in multi-label classification. Using the openly available MIMIC-III EHR dataset for phenotyping, for the first time, our work sheds light into data regimes where deep clinical models can fail to generalize. This work emphasizes the need for novel validation mechanisms driven by real-world domain shifts in AI for healthcare.

Introduction

Deep learning models for healthcare often boast impressive accuracy in controlled laboratory settings, yet they frequently stumble when deployed in real-world clinics. This performance gap stems from the "small data regime" inherent in clinical AI; even large datasets like MIMIC-III represent narrow demographic slices and specific institutional practices. When a model trained at a major urban hospital is moved to a rural clinic, it encounters domain shifts: systematic differences in population demographics (age, race), disease prevalence, or data recording protocols.

Standard validation—splitting a single dataset into training and testing sets—fails to account for these discrepancies. This paper introduces a framework to "stress-test" models by emulating common domain shifts, characterizing exactly how and why deep clinical models fail when they encounter environments they were not specifically prepared for.

Methodology and Disease Landscapes

The study utilizes a state-of-the-art ResNet-1D convolutional neural network designed for clinical time-series data from the MIMIC-III database. The dataset includes 76 clinical measurements and 25 disease categories for phenotyping tasks.

To understand the relationship between different diseases, the authors employ the Information Sieve, an information-theoretic approach that identifies latent factors explaining the correlation between variables. By mapping these "disease landscapes," the authors identify clusters of conditions that tend to co-occur.

Figure 1: Disease landscapes identified via Information Sieve, showing clusters of co-occurring conditions used to define shift scenarios.

Using these landscapes, the authors manually curate source (training) and target (testing) datasets to emulate three primary shift types:

  1. Population Biases: Training on one demographic (e.g., White patients) and testing on another (Minority groups).
  2. Label Distribution Shifts: Introducing novel disease combinations at test time (e.g., training on single diseases, testing on co-morbidities).
  3. Measurement Discrepancies: Simulating real-world noise, such as altered sampling frequencies or label errors.

Results: Quantifying Model Fragility

Performance was measured using the weighted average AUPRC (Area Under the Precision-Recall Curve). The results reveal that clinical models are surprisingly fragile when the underlying disease or demographic landscape shifts.

Figure 2: Model performance (AUPRC) across three categories of domain shifts: (a) Population Biases, (b) Label Distribution Shifts, and (c) Measurement Discrepancies.

Key Findings

  • Demographic Fragility: Shifts in race (White-to-Minority) and age (Older-to-Younger) showed the most significant drops in reliability. As seen in Figure 2(a), the "Older-to-Younger" shift resulted in an AUPRC drop to approximately 0.63, compared to 0.9 for "Younger-to-Older."
  • Contextual Learning: Models do not just learn a disease; they learn its context. When a disease appears in a new "neighborhood" of co-occurring conditions, performance tanks. In the "Single-to-Dual" label shift (Figure 2b), the model struggled to recognize co-morbidities it hadn't seen in combination during training.
  • Sampling Rate Sensitivity: While models were relatively robust to missing measurements, changes in sampling rates (e.g., checking vitals less frequently) were significantly more detrimental to generalization than simple data gaps.
  • Label Noise: Models showed resilience to 10% label noise, but performance degraded sharply once noise reached 20% (Figure 2c).

Discussion and Conclusion

This study highlights that biological "aleatoric" uncertainty—the inherent randomness in how different bodies react to disease—cannot be solved simply by using more sophisticated neural networks. The significant performance drops for minority groups and younger populations suggest that off-the-shelf deep learning is not inherently fair and requires specific algorithmic interventions.

Future work should focus on advanced temporal modeling to handle sampling rate shifts and validating these findings across disparate hospital systems. Ultimately, this research emphasizes that robust AI for healthcare requires validation mechanisms driven by real-world domain shifts rather than simple static dataset splits.

Summary updated

Abstract

The hypothesis that computational models can be reliable enough to be adopted in prognosis and patient care is revolutionizing healthcare. Deep learning, in particular, has been a game changer in building predictive models, thus leading to community-wide data curation efforts. However, due to inherent variabilities in population characteristics and biological systems, these models are often biased to the training datasets. This can be limiting when models are deployed in new environments, when there are systematic domain shifts not known a priori. In this paper, we propose to emulate a large class of domain shifts, that can occur in clinical settings, with a given dataset, and argue that evaluating the behavior of predictive models in light of those shifts is an effective way to quantify their reliability. More specifically, we develop an approach for building realistic scenarios, based on analysis of \textit{disease landscapes} in multi-label classification. Using the openly available MIMIC-III EHR dataset for phenotyping, for the first time, our work sheds light into data regimes where deep clinical models can fail to generalize. This work emphasizes the need for novel validation mechanisms driven by real-world domain shifts in AI for healthcare.
BibTeX
@misc{thiagarajan2018understanding,
  author = {Jayaraman J. Thiagarajan and Deepta Rajan and Prasanna Sattigeri},
  title = {{Understanding Behavior of Clinical Models under Domain Shifts}},
  year = {2018},
  eprint = {1809.07806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.07806},
}
Figure