Predicting adverse outcomes due to diabetes complications with machine learning using administrative health data

Mathieu Ravaut, Hamed Sadeghi, Kin Kwan Leung, Maksims Volkovs, Kathy Kornas, Vinyas Harish, Tristan Watson, Gary F. Lewis, Alanna Weisman, Tomi Poutanen, Laura C. RosellaNpj Digit. Medicine 2021 · OpenReview

TL;DR

This paper presents a machine learning model that predicts the 3-year risk of adverse diabetes complications using only administrative health data. This approach is crucial for population-level health planning and resource allocation, demonstrating the power of readily available data for proactive healthcare management.

What Problem Does This Paper Address?

Diabetes is a growing global health crisis, leading to severe complications like hyperglycemia, infections, and cardiovascular events. These complications not only impact patient quality of life but also impose a massive financial burden on healthcare systems due to hospitalizations and emergency visits. While clinical models exist for individual risk prediction, there's a significant gap in tools that can effectively predict these adverse outcomes at a population level. Such population-level insights are vital for governments and health insurance providers to plan interventions, allocate resources efficiently, and address systemic barriers to diabetes care, like socioeconomic disparities and access to healthcare.

Methods

The researchers developed a Gradient Boosting Decision Tree (GBDT) model, implemented using the XGBoost library, to predict adverse outcomes from diabetes complications. They employed a multi-instance learning approach, treating each patient's health history at different time points as separate instances. This allowed them to simulate continuous population screening. Extensive feature engineering was performed, extracting over 700 features from various administrative data sources. These features captured demographics, geographical information, chronic conditions, and healthcare utilization history. To handle the imbalanced nature of the data (adverse outcomes are rare), they used undersampling of negative instances. Model performance was evaluated using the Area Under the Receiver Operating Curve (AUC) for discrimination and calibration plots for accuracy.

Datasets

The study utilized administrative health data from Ontario, Canada, a single-payer healthcare system. This included:

  • Demographic information
  • Physician and emergency claims (Ontario Health Insurance Plan - OHIP)
  • Hospitalization history (Discharge Abstract Database - DAD)
  • Emergency services (National Ambulatory Care Reporting System - NACRS)
  • Prescription medication claims (for individuals 65+ and those receiving social assistance)
  • Diabetes-related laboratory test results (Ontario Laboratory Information System - OLIS)
  • Immigration data (Immigration Refugees and Citizenship Canada - IRCC)
  • Neighbourhood-level socioeconomic data (2001, 2006, 2011 Canadian censuses - ON-Marg)
  • Death records (Office of the Registrar General-Deaths - ORG-D)

The dataset comprised over 1.5 million patients, with data partitioned into training (1,029,366 patients), validation (272,864 patients), and testing (265,406 patients) sets.

Key Insights and Ideas

  • Administrative Data is Powerful: The study demonstrates that readily available administrative health data, often overlooked for complex predictive modeling, can be effectively leveraged to predict significant health outcomes. This opens doors for cost-effective, large-scale health management.
  • Social Determinants Matter: Beyond clinical data, features related to socio-demographic factors like immigration status and area-level ethnic concentration were found to be highly predictive. This highlights the importance of considering social determinants of health in population-level risk assessment.
  • Explainability is Key: The use of GBDT models allowed for feature contribution analysis, providing insights into why the model makes certain predictions. This interpretability is crucial for building trust and facilitating the adoption of these models in real-world healthcare planning.
  • Targeting High-Risk Individuals is Cost-Effective: The model identified that a small percentage of the highest-risk patients accounted for a disproportionately large share of the total annual cost associated with diabetes complications. This suggests that targeted interventions for these individuals could yield significant cost savings.

Key Results

  • The developed model achieved a strong average test AUC of 77.7 (range 77.7–77.9) for predicting adverse outcomes due to diabetes complications within a three-year window.
  • The model demonstrated good calibration across different population subgroups.
  • The best performance was observed for predicting hyper/hypoglycemia (AUC 84.4), while amputation had the lowest AUC (68.9).
  • The top features varied by complication, but common themes included diagnosis history, emergency department visits, and age.
  • The model could identify the top 1% of at-risk patients, who accounted for approximately 11% of the total annual cost of diabetes complications.

Limitations and Future Work

  • "2-Claim" Algorithm Limitation: The algorithm used to flag diabetes patients had a sensitivity of 86%, meaning some type 1 diabetes patients might have been missed. While a type 1 specific algorithm was considered, it would have excluded too many patients.
  • Primary Care Data Gaps: The study focused on hospitalizations and ambulatory care, not accounting for adverse events treated solely in primary care, which might be underrepresented in the data.
  • Prescription Data Limitations: Prescription data was unavailable for individuals under 65 without social assistance, potentially limiting predictive performance.
  • Causality vs. Correlation: The model identifies correlations, not causal relationships. Inferring causality would require a different analytical framework.
  • Future Work: The authors suggest external validation, recalibration, and prospective validation are crucial before widespread implementation. They also propose integrating with other data sources like EHRs and exploring causal inference frameworks.

Summary updated

Abstract

Across jurisdictions, government and health insurance providers hold a large amount of data from patient interactions with the healthcare system. We aimed to develop a machine learning-based model for predicting adverse outcomes due to diabetes complications using administrative health data from the single-payer health system in Ontario, Canada. A Gradient Boosting Decision Tree model was trained on data from 1,029,366 patients, validated on 272,864 patients, and tested on 265,406 patients. Discrimination was assessed using the AUC statistic and calibration was assessed visually using calibration plots overall and across population subgroups. Our model predicting three-year risk of adverse outcomes due to diabetes complications (hyper/hypoglycemia, tissue infection, retinopathy, cardiovascular events, amputation) included 700 features from multiple diverse data sources and had strong discrimination (average test AUC = 77.7, range 77.7–77.9). Through the design and validation of a high-performance model to predict diabetes complications adverse outcomes at the population level, we demonstrate the potential of machine learning and administrative health data to inform health planning and healthcare resource allocation for diabetes management.
BibTeX
@inproceedings{ravaut2021predicting,
  author = {Mathieu Ravaut and Hamed Sadeghi and Kin Kwan Leung and Maksims Volkovs and Kathy Kornas and Vinyas Harish and Tristan Watson and Gary F. Lewis and Alanna Weisman and Tomi Poutanen and Laura C. Rosella},
  title = {{Predicting adverse outcomes due to diabetes complications with machine learning using administrative health data}},
  booktitle = {Npj Digit. Medicine},
  year = {2021},
  url = {https://openreview.net/forum?id=xq4Z4B5HlYA},
}
Figure