Scalable and accurate deep learning with electronic health records

Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj, Michaela Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Yi Zhang, Gerardo Flores, Gavin E. Duggan, Jamie Irvine, Quoc Le, Kurt Litsch, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H. Shah, Atul J. Butte, Michael D. Howell, Claire Cui, Gregory S. Corrado, Jeffrey Dean

TL;DR

This paper introduces a novel deep learning approach that leverages the entirety of a patient's raw electronic health record (EHR), including free-text notes, to predict various clinical outcomes with high accuracy. By avoiding laborious manual feature engineering and data harmonization, this method offers a scalable and more effective alternative to traditional predictive models, paving the way for more personalized and efficient healthcare.

What Problem Does This Paper Address?

Traditional predictive modeling in healthcare relies heavily on extracting and curating specific variables from electronic health records (EHRs). This process is incredibly labor-intensive, often discarding the vast majority of valuable information within a patient's record. Furthermore, it limits the scalability of these models, as each new prediction task requires a custom dataset and significant preprocessing. This inefficiency hinders the development of accurate and widely applicable predictive tools that could significantly improve patient care, reduce costs, and enable personalized medicine. The authors aim to overcome these limitations by developing a method that can directly learn from the raw, comprehensive EHR data.

Methods

The core of this paper's innovation lies in its data representation and deep learning approach. Instead of manually engineering features, the researchers developed a pipeline to transform raw EHR data into the Fast Healthcare Interoperability Resources (FHIR) format. This standardized, hierarchical format allows for the representation of a patient's entire record, including clinical notes, without requiring site-specific harmonization.

They then employed deep learning models, specifically exploring recurrent neural networks (LSTM), attention-based time-aware neural networks (TANNs), and boosted decision stumps. These models were trained on the FHIR-formatted data, allowing them to learn representations of key factors and interactions directly from the raw input. The models were ensembled to combine the strengths of different architectures. A key aspect of their approach is the ability to process a massive volume of data, treating each piece of information (a "token") as a potential predictor.

Datasets

The study utilized de-identified EHR data from two large US academic medical centers:

  • Hospital A (University of California, San Francisco - UCSF): Data from 2012 to 2016.
  • Hospital B (University of Chicago Medicine - UCM): Data from 2009 to 2016.

These datasets included patient demographics, provider orders, diagnoses, procedures, medications, laboratory values, vital signs, flowsheet data, and de-identified free-text medical notes (from UCM). The combined dataset comprised 216,221 adult hospitalizations, representing a total of over 46 billion data points when unrolled into a temporal sequence. This is a significant dataset, encompassing a general patient population rather than being restricted to specific cohorts like ICUs.

Key Insights and Ideas

  • Leveraging the Entire EHR: The most significant insight is that deep learning models can effectively learn from the entirety of a patient's raw EHR data, including unstructured free-text notes, without the need for manual feature engineering or extensive data harmonization. This dramatically increases the amount of information available for prediction.
  • Scalability through Standardization: The use of the FHIR standard as an intermediate representation is crucial for scalability. It allows a single data processing pipeline to be applied across different healthcare systems, reducing the effort required to deploy predictive models to new sites.
  • Direct Feature Learning: Deep learning models can simultaneously learn to harmonize inputs and predict medical events, bypassing the labor-intensive preprocessing steps traditionally required. This "direct feature learning" is a paradigm shift.
  • Improved Interpretability: While deep learning models are often seen as "black boxes," the authors demonstrate that attribution methods can visualize which parts of a patient's record the model attended to, offering a degree of explainability that can build clinician trust.
  • Early Prediction Potential: The models demonstrated the ability to make accurate predictions much earlier in a patient's hospital stay than traditional methods, offering more time for clinical intervention.

Key Results

The deep learning models achieved high accuracy across multiple clinical prediction tasks, consistently outperforming traditional models:

  • In-hospital Mortality: Area Under the Receiver Operator Characteristic Curve (AUROC) of 0.93-0.94 (compared to 0.85-0.86 for traditional models). The deep learning model achieved similar accuracy approximately 24-48 hours earlier than baseline models.
  • 30-Day Unplanned Readmission: AUROC of 0.75-0.76 (compared to 0.70-0.68 for traditional models).
  • Prolonged Length of Stay (≥7 days): AUROC of 0.85-0.86 (compared to 0.76-0.74 for traditional models).
  • Discharge Diagnoses: Frequency-weighted AUROC of 0.90 (outperforming previous benchmarks).

In terms of clinical impact, for mortality prediction, the deep learning model required approximately half the number of alerts to detect high-risk patients compared to a traditional model, suggesting a significant reduction in alert fatigue.

Limitations and Future Work

The authors acknowledge several limitations:

  • Retrospective Study: The study is retrospective, and prospective trials are needed to confirm the clinical utility and impact of these predictions.
  • Generalizability to New Sites: While the models showed comparable performance across two sites, further research is needed to determine how well models trained at one site can be applied to another (transfer learning) and how to best handle variations in data and cohorts.
  • Computational Intensity: The methods are computationally intensive to develop, though running predictions on new patients is fast.
  • Incremental Contribution of Data Types: The study did not isolate the exact contribution of specific data types (e.g., clinical notes) to predictive performance, which is an area for future investigation.
  • Cohort Differences: Differences in AUROC between the two hospitals could be due to cohort differences, not just the presence or absence of notes.

Future work suggested includes demonstrating the clinical impact through prospective trials, developing better methods for transfer learning across sites, and further investigating the incremental contribution of different data modalities (like free-text notes) to predictive accuracy.

Summary updated

Abstract

Artificial intelligence outperforms traditional statistical models at predicting a range of clinical outcomes from a patient’s entire raw electronic health record (EHR). A team led by Alvin Rajkomar and Eyal Oren from Google in Mountain View, California, USA, developed a data processing pipeline for transforming EHR files into a standardized format. They then applied deep learning models to data from 216,221 adult patients hospitalized for at least 24 h each at two academic medical centers, and showed that their algorithm could accurately predict risk of mortality, hospital readmission, prolonged hospital stay and discharge diagnosis. In all cases, the method proved more accurate than previously published models. The authors provide a case study to serve as a proof-of-concept of how such an algorithm could be used in routine clinical practice in the future.
BibTeX
@inproceedings{rajkomar2018scalable,
  author = {Alvin Rajkomar and Eyal Oren and Kai Chen and Andrew M. Dai and Nissan Hajaj and Michaela Hardt and Peter J. Liu and Xiaobing Liu and Jake Marcus and Mimi Sun and Patrik Sundberg and Hector Yee and Kun Zhang and Yi Zhang and Gerardo Flores and Gavin E. Duggan and Jamie Irvine and Quoc Le and Kurt Litsch and Alexander Mossin and Justin Tansuwan and De Wang and James Wexler and Jimbo Wilson and Dana Ludwig and Samuel L. Volchenboum and Katherine Chou and Michael Pearson and Srinivasan Madabushi and Nigam H. Shah and Atul J. Butte and Michael D. Howell and Claire Cui and Gregory S. Corrado and Jeffrey Dean},
  title = {{Scalable and accurate deep learning with electronic health records}},
  booktitle = {Npj Digit. Medicine},
  year = {2018},
  url = {https://openreview.net/forum?id=j3a0Nx5n7z},
}
Figure