Vision Impairment prediction for patients diagnosed with Multiple Sclerosis: Cosmos based machine learning model training and evaluation

Brandon Buxton, Amr Hassan, Nevin Shalaby, John Lindsey, John A. Lincoln, Elmer V. Bernstam, Wagida A. Anwar, Degui Zhi, Laila RasmymedRxiv 2023 · DOI

Abstract

Multiple sclerosis (MS) is a complex autoimmune neurological disorder that frequently impacts vision. One of the most frequent initial presentations of MS is acute vision loss due to optic neuritis, an acute disorder caused by MS involvement with the optic nerve. While vision impairment is often the first sign of MS, it can occur or recur at any time during the patient’s course. In this study, we aim to develop and evaluate machine learning models to predict vision impairment in patients with MS, both at the time of first MS diagnosis and throughout their course of care. Early awareness and intervention in patients likely to have vision loss can help preserve patient quality of life. Using the Epic Cosmos de-identified electronic health record (EHR) dataset, we queried 213+ million patients to extract our MS cohort. Cases were defined as MS patients with vision impairment or optic neuritis (VI) following their first MS diagnosis, while controls were MS patients without VI. We trained logistic regression (LR), light gradient boosting machine (LGBM), and recurrent neural network (RNN) models to predict future VI in MS patients. The models were evaluated for two distinct clinical tasks: prediction of VI at the time of the first MS diagnosis and prediction of VI at the most recent visit. Similarly, we trained the models on different segments of the patient trajectory including up until the first MS diagnosis (MS-First Diagnosis), or until the most recent visit before developing the outcome (MS-Progress) as well as the combination of both (MS-General). Finally, we trained a survival model with the goal of predicting patient likelihood of vision loss over time. We compared the models’ performance using AUROC, AUPRC, and Brier scores. Our trained models achieved ∼80% AUROC, with RNN-based models outperforming LGBM and LR (79.6% vs 72.8% and 68.6%, respectively) when considering the full patient trajectory. The MS-General RNN model had the highest AUROC (64.4%) for predicting VI at the first MS diagnosis. The MS-Progress survival model achieved a 75% concordance index on the full trajectory, while the more clinically relevant MS-First Diagnosis model achieved 63.1% at initial diagnosis. RNN based models showed promising performance that demonstrates that they can be useful clinical tools to predict risk of future VI events in patients with MS.

Introduction

Multiple Sclerosis (MS) is a progressive neurological disease characterized by the immune system attacking the protective sheath of nerves. Vision impairment (VI), particularly acute vision loss from optic neuritis (ON), is one of the most common and debilitating symptoms. While vision loss is often the first sign of MS, it can recur or develop at any point during the disease course.

Current clinical tools, such as the Expanded Disability Status Scale (EDSS), often fail to capture visual complaints adequately, leading to potential underestimation of disability. Furthermore, diagnostic tools like Optical Coherence Tomography (OCT) are typically used to evaluate damage after it has occurred. There is a significant gap in the ability to predict which MS patients are at high risk for future vision loss using standard, structured electronic health record (EHR) data. This study addresses this gap by developing machine learning models to predict VI onset.

Methodology

The researchers compared three machine learning architectures to predict the onset of VI:

  1. Logistic Regression (LR): A standard baseline for binary classification.
  2. Light Gradient Boosting Machine (LGBM): A high-performance tree-based model.
  3. Recurrent Neural Networks (RNN): Specifically, a bi-directional gated-recurrent unit (Bi-GRU), chosen for its ability to handle the temporality of medical events.

Models were trained on three distinct data segments:

  • MS-First Diagnosis: Data available only up to the initial MS diagnosis.
  • MS-Progress: The full patient trajectory until the visit before a VI event.
  • MS-General: A combination of both datasets.

Additionally, an RNN-based survival model was developed to predict the likelihood of vision loss over time, categorizing patients into high, medium, and low-risk cohorts.

Datasets and Cohort Characteristics

The study utilized Epic Cosmos, a massive de-identified EHR dataset containing over 213 million patients. From this, an MS cohort was identified via SNOMED and ICD-10 codes. The primary analysis focused on 377,097 MS patients, identifying 42,281 cases who developed VI or optic neuritis after their initial MS diagnosis.

The clinical landscape of the MS cohort includes various comorbidities. As shown in Table 1, vision impairment (including optic neuritis) is a significant burden, affecting approximately 17% of the broader MS population queried in the dataset.

Table 1: Prevalence of vision impairment and other comorbidities in the MS study cohort.

Key Results

The study demonstrated that the sequence of medical events is a critical predictor for MS progression.

  • Top Performance: RNN-based models achieved an AUROC of 79.6% when considering the full patient trajectory, significantly outperforming LGBM (72.8%) and LR (68.6%).
  • Precision-Recall: The RNN achieved an AUPRC of 40.5%, roughly 30% higher than the base prevalence of vision loss in the cohort.
  • Survival Prediction: The MS-Progress survival model achieved a 75% concordance index on the full trajectory.
  • Initial Diagnosis Challenge: Predicting VI at the exact moment of first MS diagnosis proved more difficult, with the MS-General RNN model reaching an AUROC of 64.4%.

Discussion and Conclusion

The superior performance of RNN models suggests that the "gestalt" of a patient's medical history, captured through temporal data, is vital for predicting disease progression. Despite lacking "gold standard" data like MRI images or clinical notes, structured EHR data alone proved robust enough to build predictive models.

The survival model's strong discrimination at the 5-year mark suggests these models could function as early warning systems, allowing clinicians to initiate more aggressive monitoring or therapy for high-risk patients. Future work will focus on expanding these models to predict other MS-related comorbidities and validating them against international registries to ensure global generalizability.

Summary updated

Abstract

Abstract Objectives Multiple sclerosis (MS) is a complex autoimmune neurological disorder that frequently impacts vision. One of the most frequent initial presentations of MS is acute vision loss due to optic neuritis, an acute disorder caused by MS involvement with the optic nerve. While vision impairment is often the first sign of MS, it can occur or recur at any time during the patient’s course. In this study, we aim to develop and evaluate machine learning models to predict vision impairment in patients with MS, both at the time of first MS diagnosis and throughout their course of care. Early awareness and intervention in patients likely to have vision loss can help preserve patient quality of life. Materials and Methods Using the Epic Cosmos de-identified electronic health record (EHR) dataset, we queried 213+ million patients to extract our MS cohort. Cases were defined as MS patients with vision impairment or optic neuritis (VI) following their first MS diagnosis, while controls were MS patients without VI. We trained logistic regression (LR), light gradient boosting machine (LGBM), and recurrent neural network (RNN) models to predict future VI in MS patients. The models were evaluated for two distinct clinical tasks: prediction of VI at the time of the first MS diagnosis and prediction of VI at the most recent visit. Similarly, we trained the models on different segments of the patient trajectory including up until the first MS diagnosis (MS-First Diagnosis), or until the most recent visit before developing the outcome (MS-Progress) as well as the combination of both (MS-General). Finally, we trained a survival model with the goal of predicting patient likelihood of vision loss over time. We compared the models’ performance using AUROC, AUPRC, and Brier scores. Results We extracted a cohort of 377,097 patients with MS, including 42,281 VI cases. Our trained models achieved ∼80% AUROC, with RNN-based models outperforming LGBM and LR (79.6% vs 72.8% and 68.6%, respectively) when considering the full patient trajectory. The MS-General RNN model had the highest AUROC (64.4%) for predicting VI at the first MS diagnosis. The MS-Progress survival model achieved a 75% concordance index on the full trajectory, while the more clinically relevant MS-First Diagnosis model achieved 63.1% at initial diagnosis. Discussion and Conclusion The MS-Progress and MS-General RNN models performed best in both prediction scenarios. While MS-General achieved the best performance at the time of first MS diagnosis with around 1% AUROC increase compared to the MS-First Diagnosis model, it showed around 1% AUROC decrease on the MS progress scenario. All RNN survival models performed the best when they were trained on data corresponding to the evaluation use-case scenarios. RNN based models showed promising performance that demonstrates that they can be useful clinical tools to predict risk of future VI events in patients with MS. Further development of these models will focus on expanding to predict other comorbidities associated with MS relapse or progression.
BibTeX
@misc{buxton2023vision,
  author = {Brandon Buxton and Amr Hassan and Nevin Shalaby and John Lindsey and John A. Lincoln and Elmer V. Bernstam and Wagida A. Anwar and Degui Zhi and Laila Rasmy},
  title = {{Vision Impairment prediction for patients diagnosed with Multiple Sclerosis: Cosmos based machine learning model training and evaluation}},
  year = {2023},
  doi = {10.1101/2023.11.10.23298366},
  howpublished = {medRxiv},
}
Figure