TL;DR
This paper introduces "pan-diagnostic machine learning" to predict the risk of thousands of diseases simultaneously from electronic health records (EHRs). They demonstrate that this approach can achieve high predictive performance across a broad range of diagnoses, providing a comprehensive patient risk profile and a valuable new dataset for future research.
What Problem Does This Paper Address?
While machine learning has shown promise in predicting the risk of specific diseases (like breast cancer or heart attacks) using EHR data, there's a significant knowledge gap regarding how well most diseases can be predicted and how far in advance. The current approach of building individual models for each disease is fragmented and doesn't offer a holistic view of a patient's health risks. This paper aims to bridge this gap by developing a scalable method to predict risks for thousands of diagnosis codes concurrently, establishing a baseline for diagnostic predictability across the board.
Methods
The core innovation is a "high-throughput machine learning" pipeline designed to build thousands of individual predictive models for each diagnosis code. This pipeline uses random forests for their accuracy and resilience to high-dimensional data. A key challenge addressed is automatically defining appropriate "case" (patients who received a diagnosis) and "control" (similar patients who did not) groups for each specific diagnosis. This involves a novel "dynamic definition refinement" (DDR) process to identify prerequisite diagnoses that should be present in both cases and controls, preventing artificially high accuracy for conditions that are complications of others (e.g., predicting diabetic complications without considering diabetes itself). The models are trained on EHR data and evaluated using Area Under the Receiver Operating Characteristic Curve (AUC) via ten-fold cross-validation. They also conducted a simulated prospective study to assess real-world translation.
Datasets
The primary dataset used is from the [REDACTED_FOR_REVIEW] Health System (RFRHS), comprising de-identified demographics, diagnoses (ICD-9 codes), labs, vitals, and procedures for approximately 1.1 million patients. This dataset is used to train and evaluate the pan-diagnostic models. The authors also introduce a new dataset containing the AUC scores and feature importance values for the 24,482 predictive models generated, intended as a benchmark for future research.
Key Insights and Ideas
- Pan-diagnostic prediction is feasible: The core hypothesis that nearly all disease risks can be predicted simultaneously at a competitive level is supported.
- Time horizon matters: Predictive accuracy significantly decreases as the prediction window extends further into the future, which is intuitive but now quantified across thousands of diseases.
- Disease categories have varying predictability: Some disease categories are much easier to predict than others, suggesting underlying differences in their nature or the data available to capture them.
- Feature importance reveals known risk factors: Even with a high-throughput approach, the models often identify features that align with known clinical risk factors for specific diseases, lending credibility to the approach.
- The "rule of two" for cases: Requiring a diagnosis to appear at least twice in a patient's record for it to be considered a "case" helps ensure a more robust definition of the event of interest.
Key Results
- The pipeline successfully built models for thousands of ICD-9 diagnosis codes.
- Average AUCs for predicting diagnoses 1 month in advance were 0.803, and for 6 months in advance, 0.758.
- Predictive performance generally decreases with longer prediction windows, with average AUCs for 20-year predictions dropping to 0.524.
- Despite the broad scope, some diagnoses, particularly ocular disorders and congenital anomalies, showed predictive power even 20 years in advance.
- The simulated prospective study showed slightly lower performance (mean AUC of 0.697 for a 1-year window) compared to the initial cross-validation experiments, highlighting potential challenges in real-world translation.
Limitations and Future Work
- Data-driven truncation windows: The paper acknowledges the lack of a data-driven method to automatically determine the optimal prediction window (truncation date) for each specific diagnosis.
- Comparison to clinical practice: The study does not compare its models' performance against current clinical practice or human expert predictions, making it difficult to assess the immediate clinical impact.
- De-identified data: The use of de-identified coded data for privacy reasons limits the inclusion of rich textual information or more advanced phenotyping methods that could potentially improve model accuracy.
- Future work: The authors suggest incorporating existing disease ontologies to group ICD-9 codes for more efficient and disease-specific models, exploring multi-label learning approaches, and conducting clinical trials to validate the models in real-world settings. They also emphasize the value of the released dataset for further research into diagnosis clustering and predictive relationships.
Summary updated