Predicting the risk of diabetes complications using machine learning and social administrative data in a country with ethnic inequities in health: Aotearoa New Zealand

Nhung Nghiem, Nick Wilson, Jeremy Krebs, Truyen Tran

TL;DR

Researchers in Aotearoa New Zealand have demonstrated that machine learning (ML) can accurately predict cardiovascular disease (CVD) risk in diabetic populations using only "social administrative data"—national records like census info and hospitalizations—rather than traditional clinical markers. Crucially, the study found that high-performing models like Xgboost do not require a trade-off between overall predictive accuracy and ethnic equity, suggesting that population-level health planning can be both precise and fair.

What Problem Does This Paper Address?

Cardiovascular disease is the leading cause of death globally, and those living with diabetes face a significantly higher risk. In New Zealand, this burden is not shared equally; Māori, Pasifika, and Asian populations face major health inequities driven by socio-economic and systemic factors.

While many CVD risk prediction models exist (like the NZ PREDICT equations), they typically rely on "rich" clinical data—think BMI, blood pressure, and specific biomarkers. These are great for a doctor's office but difficult to deploy at a national scale for population health planning. The authors sought to determine if "administrative data" (records already collected by the government for non-clinical purposes) could be used to predict complications and whether these models would remain fair across different ethnic groups.

Methods

The researchers compared five different ML architectures:

  1. Logistic Regression (The baseline)
  2. Decision Trees
  3. Random Forest (RF)
  4. Neural Networks
  5. Xgboost (Extreme Gradient Boosting)

What makes their approach novel is the Optimization Scenarios. They didn't just train one model; they tested three strategies to see how to best handle ethnicity:

  • Scenario A: Train and optimize on the whole population.
  • Scenario B: Train on the whole population but optimize the hyperparameters specifically for a single ethnic group.
  • Scenario C: Train and optimize only on data from a specific ethnic group.

They used an 80/20 train/test split and validated the models against a future time period (2016–2018) to ensure the predictions held up over time.

Datasets

The study utilized the Stats NZ Integrated Data Infrastructure (IDI), a massive, linked research database. Specific datasets included:

  • Census 2013: For smoking status, language, and employment.
  • Ministry of Health Chronic Condition Table: To identify individuals with diabetes and other comorbidities.
  • Pharmaceutical Data: To track preventive medications.
  • IDI Population Explorer: For social indicators like social housing use and major life events (e.g., divorce).

The final cohort included approximately 146,000 New Zealand residents aged 30–74 with known diabetes but no prior history of CVD.

Key Insights and Ideas

  • Administrative Data is a Viable Proxy: Even without blood pressure readings or BMI, the models achieved an Area Under the Curve (AUC) of 0.74. This is comparable to models that do use clinical data, proving that social determinants and history of service use are powerful predictors of health outcomes.
  • The "Fairness" Win-Win: One of the most significant findings was that there was no trade-off between model performance and equity. Improving the model for the whole population generally improved it for sub-groups, and vice versa.
  • Bigger is Better for Training: Interestingly, Scenario C (training on specific ethnic groups) performed worse than using the whole population. This suggests that the "shared" patterns of disease progression across the entire population provide more signal than the specific "unique" patterns within a smaller ethnic subset.
  • Non-Clinical Variables Matter: "Geographical area" and "Income level" appeared in the top 10 most important variables for the Random Forest model, yet these are almost never included in traditional clinical risk calculators.

Key Results

  • Top Performer: Xgboost was the clear winner, outperforming Random Forest by about 6.4% for the total population and up to 10% for the Pasifika population.
  • Predictive Power: The Xgboost model achieved an average AUC of 0.74 for predicting CVD events three years into the future.
  • Variable Importance: The top predictors were Age, Geographical area, Hospitalization history, and Years living with diabetes.
  • Stability: The models remained stable when tested on the validation dataset (2016–2018), showing that the patterns the ML learned weren't just "noise" from a specific year.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Missing Clinical Data: While the model performed well without it, they expect that adding clinical features (like HbA1c levels) would further boost accuracy.
  • Computational Barriers: Running these models on massive, government-held datasets requires significant computing infrastructure that isn't always accessible for routine health planning.
  • Ethnicity Misclassification: There is a known risk of Māori being misclassified as non-Māori in administrative data, which could skew equity results.

Future Work: The researchers suggest expanding the feature set to investigate "unknown" CVD risk factors and applying this same methodology to other chronic conditions like cancer or kidney disease. They also emphasize that the goal isn't just better math, but better resource allocation to the groups the model identifies as being at the highest risk.

Summary updated

Abstract

ABSTRACT Background In the age of big data, linked social and administrative health data in combination with machine learning (ML) is being increasingly used to improve prediction in cardiovascular diseases (CVD). We aimed to apply ML methods on extensive national-level health and social administrative datasets to predict future diabetes complications by ethnicity. Methods Five ML models were used to predict CVD events among all people with known diabetes in the population of New Zealand, utilizing national-level administrative data at the individual level. Results The Xgboost ML model had the best predictive power for predicting CVD events three years into the future among the population with diabetes. The optimization procedure also found limited improvement in AUC by ethnicity. The results indicated no trade-off between model predictive performance and equity gap of prediction by ethnicity. The list of variables of importance was different among different models/ethnic groups, for examples: age, deprivation, having had a hospitalization event, and the number of years living with diabetes. Discussion and conclusions We provide further evidence that ML with administrative health data can be used for meaningful future prediction of health outcomes. As such it could be utilized to inform health planning and healthcare resource allocation for diabetes management and the prevention of CVD events. Our results may suggest limited scope for developing prediction models by ethnic group and that the major ways to reduce inequitable health outcomes is probably via improved delivery of prevention and management to those groups with diabetes at highest need.
BibTeX
@article{nghiem2024predicting,
  author = {Nhung Nghiem and Nick Wilson and Jeremy Krebs and Truyen Tran},
  title = {{Predicting the risk of diabetes complications using machine learning and social administrative data in a country with ethnic inequities in health: Aotearoa New Zealand}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2024},
  doi = {10.1186/s12911-024-02678-x},
}
Figure