High-Throughput Machine Learning from Electronic Health Records

Ross S. Kleiman, Paul S. Bennett, Peggy L. Peissig, Richard L. Berg, Zhaobin Kuang, Scott J. Hebbring, Michael D. Caldwell, David PagearXiv 2019 · OpenReview

TL;DR

This paper introduces "pan-diagnostic machine learning" to predict the risk of thousands of diseases simultaneously from electronic health records (EHRs). They demonstrate that this approach can achieve high predictive performance across a broad range of diagnoses, providing a comprehensive patient risk profile and a valuable new dataset for future research.

What Problem Does This Paper Address?

While machine learning has shown promise in predicting the risk of specific diseases (like breast cancer or heart attacks) using EHR data, there's a significant knowledge gap regarding how well most diseases can be predicted and how far in advance. The current approach of building individual models for each disease is fragmented and doesn't offer a holistic view of a patient's health risks. This paper aims to bridge this gap by developing a scalable method to predict risks for thousands of diagnosis codes concurrently, establishing a baseline for diagnostic predictability across the board.

Methods

The core innovation is a "high-throughput machine learning" pipeline designed to build thousands of individual predictive models for each diagnosis code. This pipeline uses random forests for their accuracy and resilience to high-dimensional data. A key challenge addressed is automatically defining appropriate "case" (patients who received a diagnosis) and "control" (similar patients who did not) groups for each specific diagnosis. This involves a novel "dynamic definition refinement" (DDR) process to identify prerequisite diagnoses that should be present in both cases and controls, preventing artificially high accuracy for conditions that are complications of others (e.g., predicting diabetic complications without considering diabetes itself). The models are trained on EHR data and evaluated using Area Under the Receiver Operating Characteristic Curve (AUC) via ten-fold cross-validation. They also conducted a simulated prospective study to assess real-world translation.

Datasets

The primary dataset used is from the [REDACTED_FOR_REVIEW] Health System (RFRHS), comprising de-identified demographics, diagnoses (ICD-9 codes), labs, vitals, and procedures for approximately 1.1 million patients. This dataset is used to train and evaluate the pan-diagnostic models. The authors also introduce a new dataset containing the AUC scores and feature importance values for the 24,482 predictive models generated, intended as a benchmark for future research.

Key Insights and Ideas

  • Pan-diagnostic prediction is feasible: The core hypothesis that nearly all disease risks can be predicted simultaneously at a competitive level is supported.
  • Time horizon matters: Predictive accuracy significantly decreases as the prediction window extends further into the future, which is intuitive but now quantified across thousands of diseases.
  • Disease categories have varying predictability: Some disease categories are much easier to predict than others, suggesting underlying differences in their nature or the data available to capture them.
  • Feature importance reveals known risk factors: Even with a high-throughput approach, the models often identify features that align with known clinical risk factors for specific diseases, lending credibility to the approach.
  • The "rule of two" for cases: Requiring a diagnosis to appear at least twice in a patient's record for it to be considered a "case" helps ensure a more robust definition of the event of interest.

Key Results

  • The pipeline successfully built models for thousands of ICD-9 diagnosis codes.
  • Average AUCs for predicting diagnoses 1 month in advance were 0.803, and for 6 months in advance, 0.758.
  • Predictive performance generally decreases with longer prediction windows, with average AUCs for 20-year predictions dropping to 0.524.
  • Despite the broad scope, some diagnoses, particularly ocular disorders and congenital anomalies, showed predictive power even 20 years in advance.
  • The simulated prospective study showed slightly lower performance (mean AUC of 0.697 for a 1-year window) compared to the initial cross-validation experiments, highlighting potential challenges in real-world translation.

Limitations and Future Work

  • Data-driven truncation windows: The paper acknowledges the lack of a data-driven method to automatically determine the optimal prediction window (truncation date) for each specific diagnosis.
  • Comparison to clinical practice: The study does not compare its models' performance against current clinical practice or human expert predictions, making it difficult to assess the immediate clinical impact.
  • De-identified data: The use of de-identified coded data for privacy reasons limits the inclusion of rich textual information or more advanced phenotyping methods that could potentially improve model accuracy.
  • Future work: The authors suggest incorporating existing disease ontologies to group ICD-9 codes for more efficient and disease-specific models, exploring multi-label learning approaches, and conducting clinical trials to validate the models in real-world settings. They also emphasize the value of the released dataset for further research into diagnosis clustering and predictive relationships.

Summary updated

Abstract

The widespread digitization of patient data via electronic health records (EHRs) has created an unprecedented opportunity to use machine learning algorithms to better predict disease risk at the patient level. Although predictive models have previously been constructed for a few important diseases, such as breast cancer and myocardial infarction, we currently know very little about how accurately the risk for most diseases or events can be predicted, and how far in advance. Machine learning algorithms use training data rather than preprogrammed rules to make predictions and are well suited for the complex task of disease prediction. Although there are thousands of conditions and illnesses patients can encounter, no prior research simultaneously predicts risks for thousands of diagnosis codes and thereby establishes a comprehensive patient risk profile. Here we show that such pandiagnostic prediction is possible with a high level of performance across diagnosis codes. For the tasks of predicting diagnosis risks both 1 and 6 months in advance, we achieve average areas under the receiver operating characteristic curve (AUCs) of 0.803 and 0.758, respectively, across thousands of prediction tasks. Finally, our research contributes a new clinical prediction dataset in which researchers can explore how well a diagnosis can be predicted and what health factors are most useful for prediction. For the first time, we can get a much more complete picture of how well risks for thousands of different diagnosis codes can be predicted.
BibTeX
@misc{kleiman2019high,
  author = {Ross S. Kleiman and Paul S. Bennett and Peggy L. Peissig and Richard L. Berg and Zhaobin Kuang and Scott J. Hebbring and Michael D. Caldwell and David Page},
  title = {{High-Throughput Machine Learning from Electronic Health Records}},
  year = {2019},
  eprint = {1907.01901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.01901},
}
Figure