Scalable and accurate deep learning for electronic health records.

Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj, Peter J. Liu, Xiaobing Liu, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Gavin E. Duggan, Gerardo Flores, Michaela Hardt, Jamie Irvine, Quoc V. Le, Kurt Litsch, Jake Marcus, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H. Shah, Atul J. Butte, Michael Howell, Claire Cui, Greg Corrado, Jeff Dean

Abstract

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two U.S. academic medical centers with 216,221 adult patients hospitalized for at least 24 hours. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting in-hospital mortality (AUROC across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed state-of-the-art traditional predictive models in all cases. We also present a case-study of a neural-network attribution system, which illustrates how clinicians can gain some transparency into the predictions.

Introduction

Predictive modeling in healthcare is notoriously difficult to scale. Traditionally, researchers must spend months manually selecting, cleaning, and "harmonizing" specific variables from a hospital's database. This "data janitor" work often consumes 80% of the effort and discards the vast majority of available information—especially the rich, unstructured data found in clinical notes. Furthermore, most existing models only use a handful of variables, leading to imprecise predictions that cause "alert fatigue" among staff. This study sought to create a system that could ingest all data in a patient's record without site-specific customization.

Methods

The researchers moved away from manual feature engineering in favor of a "representation learning" approach. Raw EHR data from different hospitals were mapped into the Fast Healthcare Interoperability Resources (FHIR) format, creating a consistent, chronological timeline of every event. Every piece of data—medication orders, lab results, and individual words in clinical notes—was treated as a "token" in a sequence. The system utilized an ensemble of deep learning architectures, including Long Short-Term Memory (LSTM) and Time-Aware Neural Networks (TANN), to learn which features were important directly from the raw data.

Data and Scale

The study utilized de-identified EHR data from the University of California, San Francisco (UCSF) and the University of Chicago Medicine (UCM), totaling 216,221 hospitalizations and 46.8 billion data points. Unlike many previous studies, this included the full general patient population, not just those in the ICU.

Distribution of data volume (tokens) per patient at different prediction timepoints for Hospital A and Hospital B.

The model thrived on the volume and variety of data. By including free-text notes, the model could identify subtle clinical nuances that structured data might miss. The neural network learned semantic relationships automatically, proving that site-specific data harmonization is not required for high performance.

Key Results

The deep learning models significantly outperformed traditional, clinically validated baselines across all tasks. For in-hospital mortality, the models achieved an AUROC of 0.93–0.94, compared to 0.85 for the traditional augmented Early Warning Score (aEWS). For mortality alerts, the deep learning model produced half the number of false positives compared to traditional systems.

Comparison of predictive performance (AUROC) between deep learning models and traditional baselines over time at Hospital A.

Crucially, the deep learning models achieved the same level of accuracy as traditional models 24 to 48 hours earlier in the patient's stay. This increased lead time is critical for clinical intervention. Additionally, the model could predict the entire suite of a patient's final discharge diagnoses with a weighted AUROC of 0.90.

Discussion and Limitations

While the results are promising, the study was retrospective; prospective clinical trials are needed to see if these predictions improve patient outcomes in real-time. Training these models also requires significant specialized hardware and expertise. To address the "black box" nature of neural networks, the researchers used attention mechanisms to visualize which parts of a patient's chart the model prioritized, providing a proof-of-concept for clinical interpretability. Future work will explore how well a model trained at one site transfers its knowledge to a completely new institution.

Summary updated

Abstract

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two U.S. academic medical centers with 216,221 adult patients hospitalized for at least 24 hours. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting in-hospital mortality (AUROC across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed state-of-the-art traditional predictive models in all cases. We also present a case-study of a neural-network attribution system, which illustrates how clinicians can gain some transparency into the predictions. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios, complete with explanations that directly highlight evidence in the patient's chart.
BibTeX
@inproceedings{rajkomar2018scalable,
  author = {Alvin Rajkomar and Eyal Oren and Kai Chen and Andrew M. Dai and Nissan Hajaj and Peter J. Liu and Xiaobing Liu and Mimi Sun and Patrik Sundberg and Hector Yee and Kun Zhang and Gavin E. Duggan and Gerardo Flores and Michaela Hardt and Jamie Irvine and Quoc V. Le and Kurt Litsch and Jake Marcus and Alexander Mossin and Justin Tansuwan and De Wang and James Wexler and Jimbo Wilson and Dana Ludwig and Samuel L. Volchenboum and Katherine Chou and Michael Pearson and Srinivasan Madabushi and Nigam H. Shah and Atul J. Butte and Michael Howell and Claire Cui and Greg Corrado and Jeff Dean},
  title = {{Scalable and accurate deep learning for electronic health records.}},
  booktitle = {Npj Digit. Medicine},
  year = {2018},
  url = {https://openreview.net/forum?id=j3a0Nx5n7z},
}
Figure