Abstract
Applications of machine learning in healthcare are of high interest and have the potential to significantly improve patient care. Yet, the real-world accuracy and performance of these models on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate different methods that predict healthcare outcomes. To overcome patient privacy concerns, we employed a Model-to-Data approach, allowing citizen scientists and researchers to train and evaluate machine learning models on private health data without direct access to that data. We focused on the prediction of all-cause mortality as the community challenge question. In total, we had 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries. The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI 0.942, 0.951) and an area under the precision-recall curve of 0.487 (95% CI 0.458, 0.499) on patients prospectively collected over a one year observation of a large health system. Post-hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data and have similar accuracy on the population. This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
Introduction
Machine learning (ML) in healthcare faces significant barriers regarding patient privacy and model reliability. HIPAA regulations restrict access to large-scale Electronic Health Record (EHR) datasets, while "self-assessment bias" often leads to inflated performance reports on static datasets. This paper presents a framework for the EHR DREAM Challenge, evaluating how AI performs when faced with shifting real-world data and diverse patient subpopulations.
Methods: The Model-to-Data (MTD) Framework
To overcome privacy hurdles, the researchers implemented a Model-to-Data (MTD) architecture. Participants were provided with synthetic data to build and debug models locally. Final models were submitted as Docker containers and executed within a secure environment at the University of Washington. This allowed models to be trained and evaluated on private EHR data without exposing patient identifiers to the developers.
Datasets and Study Population
The study utilized the UW Medicine enterprise data warehouse, containing records for 1.3 million patients spanning a decade. Data was standardized using the OMOP Common Data Model (CDM). The population demographics across the various phases of the challenge—Leaderboard, Validation, and Post-Challenge—are summarized in the following table.
Results
The challenge saw participation from 25 independent teams. The top-performing model achieved an AUROC of 0.947 (95% CI 0.942, 0.951) and an AUPRC of 0.487 (95% CI 0.458, 0.499) on a prospectively collected validation set.
Key findings included:
- Performance Decay: A significant drop in average AUROC (0.069) was observed between the leaderboard and validation phases, likely due to covariate drift over time. The winning team mitigated this by prioritizing the most recent six months of data.
- Subpopulation Disparities: Post-hoc analysis revealed that models with similar aggregate performance exhibited varying accuracy across racial and gender groups. Notably, all models demonstrated higher accuracy for female patients compared to male patients.
- Methodological Success: Top teams favored gradient-boosted methods (e.g., LightGBM, XGBoost) and employed "ontology roll-ups" to consolidate granular medical codes into robust features.
Discussion
The EHR DREAM Challenge demonstrates the feasibility of evaluating clinical AI through a secure MTD framework. However, limitations such as computational constraints (10-hour runtime limits) and single-site data focus suggest areas for improvement. Future efforts should prioritize multi-site validation and the prediction of more clinically actionable outcomes, such as sepsis or readmission risk, to ensure AI tools are both reliable and useful in practice.
Summary updated