TL;DR
This paper tackles the challenge of predicting long-term health outcomes using electronic health records (EHRs) when long-term outcome labels are scarce for current patients. They propose a novel adversarial domain adaptation framework that aligns historical (fully labeled) data with contemporary (partially labeled) data, outperforming existing methods and nearing oracle performance.
What Problem Does This Paper Address?
Predicting long-term clinical outcomes is crucial for early intervention and improved patient care. However, electronic health records (EHRs) often lack sufficient long-term follow-up data for contemporary patient cohorts. This is exacerbated by the dynamic nature of clinical practice, meaning models trained on older, historical data might not perform well on current patient populations. The authors frame this as a "positive-unlabeled" (PU) domain adaptation problem: we have a fully labeled "source" domain (historical data) and a "target" domain (contemporary data) where outcomes are only partially observed (some positives are known, but unobserved outcomes are treated as negative, even if they might be positive). This is a significant challenge because directly applying models trained on historical data to current data can lead to poor performance due to "population shift" – differences in patient characteristics or clinical practices over time.
Methods
The core of the proposed solution is an adversarial domain adaptation framework specifically designed for PU data. It builds upon the Adversarial Discriminative Domain Adaptation (ADDA) approach but introduces three key alignment components:
- Overall Alignment: This component uses a discriminator to align the general feature distributions between the source and target domains. It aims to make the target domain's feature space resemble the source domain's, reducing overall domain discrepancy.
- Partial Alignment: This is a crucial addition for PU settings. It specifically aligns the negative samples from the source domain with the unlabeled samples from the target domain. This is important because unlabeled target samples might actually be positive, and forcing them to perfectly match source negatives could be detrimental. This component uses a KL-divergence loss to encourage alignment while allowing the support of unlabeled target samples (which might include positives) to extend beyond source negatives.
- Conditional Alignment: This component leverages the observed positive labels in the target domain. It uses a supervised loss to ensure that the model correctly predicts these known positives, helping to align the conditional distribution P(Y|X) and mitigate conditional shift.
The framework involves pre-training a source encoder and classifier, then using adversarial training with two discriminators to align features from the target encoder.
Datasets
The authors evaluate their method on:
- SVHN-MNIST: A benchmark digit classification task. SVHN (Street View House Numbers) is used as the source domain, and a stylized variant of MNIST (MNIST-C) is used as the target domain. This serves as a proof-of-concept for domain adaptation with PU data.
- Two real-world EHR datasets:
- One-year Mortality Prediction Post COVID-19: Data from Duke University Health System (DUHS) is used to predict one-year mortality after emergency department visits. A historical cohort (2018) serves as the source, and a contemporary cohort (2021) as the target, with simulated partial observability of outcomes in the target.
- Long-term Prediction of Neurodevelopmental Conditions (NDC): Also using DUHS data, this application predicts NDC diagnoses by age 5.5 years based on clinical features at 18 months. A historical cohort (children born in 2014) is the source, and a contemporary cohort (children born in 2017) is the target, with simulated partial outcome observability.
Key Insights and Ideas
- PU Domain Adaptation is Key for EHRs: The paper highlights that standard domain adaptation methods are insufficient for EHR data due to the common issue of partially observed long-term outcomes. The PU formulation is essential.
- The Power of Three Alignments: The combination of overall, partial, and conditional alignment is critical. While overall alignment reduces general domain shift, partial alignment specifically handles the uncertainty of unlabeled target samples, and conditional alignment leverages known positives to correct for conditional shifts.
- Balancing Components is Crucial: The ablation studies suggest that while each component contributes, their combination is most effective. Conditional alignment, in particular, significantly boosts performance, but it needs to be balanced with other components to avoid overfitting to the limited positive labels.
- Near-Oracle Performance is Achievable: The most striking insight is that their proposed method can achieve performance comparable to, and sometimes even exceeding, an "oracle" model trained on fully observed target data, even with significant label scarcity.
Key Results
- Digits Classification: The proposed method achieved the highest AUROC on the SVHN-MNIST task, significantly outperforming baseline models, even with very few labeled samples in the target domain.
- One-year Mortality Prediction: For predicting one-year mortality post-COVID-19, the method achieved an AUROC of 0.909 with only 90 days of outcome data, which was close to and slightly above the oracle model's AUROC of 0.907.
- Long-term NDC Prediction: In the NDC prediction task, the proposed method achieved an AUROC of 0.813 with only 6 months of outcome data, nearly matching the oracle model's 0.818.
- Ablation Studies: The results consistently showed that the full model (all three alignment components) performed best across various scenarios. Conditional alignment was particularly impactful, especially when label observability was low.
- Computational Cost: The additional training complexity from adversarial components was manageable, with total training times remaining reasonable.
Limitations and Future Work
- Calibration Error: The authors note that conditional alignment, while beneficial, can lead to larger calibration errors. They suggest that additional re-calibration strategies might be needed.
- Hyperparameter Tuning: Balancing the weights of the three alignment components (λ₁, λ₂, λ₃) is non-trivial. Due to the lack of fully observed labels in the target domain, cross-validation for tuning these parameters was not straightforward, and fixed values were used.
- Future Work: The authors suggest exploring hyperparameter searching in PU settings and further investigating re-calibration strategies. They also envision their framework being integrated into existing clinical workflows to leverage external data and address observability challenges in EHR-based predictions.
Summary updated