Comparison of machine-learning algorithms to build a predictive model for detecting undiagnosed diabetes - ELSA-Brasil: accuracy study

André Rodrigues Olivera, Valter Roesler, Cirano Iochpe, María Inês Schmidt, Álvaro Vigo, Sandhi Maria Barreto, Bruce Bartholow DuncanSao Paulo Medical Journal 2017 · DOI

TL;DR

This study compares various machine learning algorithms (logistic regression, neural networks, Naive Bayes, K-nearest neighbor, random forest) to build predictive models for detecting undiagnosed Type 2 diabetes using data from the ELSA-Brasil cohort. The findings demonstrate that readily available clinical data can be used to identify individuals at high risk, with logistic regression and artificial neural networks showing the best performance.

What Problem Does This Paper Address?

Type 2 diabetes is a significant global health issue, characterized by its chronic nature and association with severe complications. A major challenge is that a substantial portion of individuals with diabetes remain undiagnosed, exacerbating the disease's impact. Early detection and intervention are crucial for managing the condition and preventing its long-term consequences. While clinical prediction rules exist, this paper explores the potential of modern machine learning techniques to develop more accurate and easily applicable models for identifying individuals with undiagnosed diabetes, using a comprehensive Brazilian cohort.

Methods

The researchers employed a rigorous four-step process to develop and validate predictive models:

  1. Parameter Tuning: Each machine learning algorithm was optimized by testing various parameter configurations using tenfold cross-validation, repeated three times.
  2. Automatic Variable Selection: A wrapper strategy with forward selection was used to identify the most predictive subsets of variables for four of the algorithms (logistic regression, artificial neural network, K-nearest neighbor, and Naive Bayes). This step aimed to simplify models and potentially improve performance.
  3. Error Estimation: The performance of models built with the selected parameters and variable subsets (as well as the original set of variables) was estimated using tenfold cross-validation, repeated ten times for greater reliability.
  4. Generalization Testing: The best-performing model for each algorithm was evaluated on an independent test dataset to assess its ability to generalize to unseen data.

The study compared logistic regression, artificial neural networks (multilayer perceptron/backpropagation), Naïve Bayes, K-nearest neighbor, and random forest algorithms. Data preparation involved binarization and discretization of variables.

Datasets

The study utilized baseline data from the Longitudinal Study of Adult Health (ELSA-Brasil), a multicenter cohort study of Brazilian public servants aged 35-74. The initial dataset comprised 15,105 participants. After exclusions for self-reported diabetes, missing data, and other criteria, the final analysis included 12,447 participants, of whom 1,359 (11.0%) had undiagnosed diabetes. The dataset was split into a training/validation set (70%) and a test set (30%) for model development and generalization testing, respectively. No new benchmark datasets were introduced; the study leveraged existing cohort data.

Key Insights and Ideas

  • Machine Learning's Potential for Diabetes Detection: The study strongly supports the utility of machine learning algorithms in identifying individuals at high risk of undiagnosed diabetes, even with relatively simple clinical data.
  • Logistic Regression and ANNs Shine: Artificial neural networks and logistic regression consistently emerged as the top performers across various evaluation metrics, suggesting their robustness for this task.
  • Variable Selection Matters, But Simplicity is Key: While automatic variable selection helped identify smaller, more manageable sets of predictors, the performance gains were often marginal. This highlights the trade-off between model complexity and practical applicability.
  • Generalizability is Crucial: The generalization testing step is vital, as it simulates real-world application. The study demonstrates that models can maintain good performance even when applied to data not used during training.
  • Clinical Data Sufficiency: The success of models built with easily obtainable clinical and behavioral data underscores the feasibility of implementing such tools in primary care settings.

Key Results

  • The best models were developed using artificial neural networks and logistic regression.
  • In the error estimation step, these algorithms achieved mean Area Under the Curve (AUC) of 75.24% (ANN) and 74.98% (logistic regression).
  • In the generalization testing step, the AUCs were 74.17% (ANN) and 74.41% (logistic regression).
  • The logistic regression model, chosen for its interpretability and ease of application, achieved 68% sensitivity and 67.2% specificity in identifying positive cases (undiagnosed diabetes) among individuals who did not know they had the condition.
  • The study developed a prototype web tool based on the logistic regression model for practical application.

Limitations and Future Work

  • Model Interpretability: While ANNs performed well, their "black box" nature can make interpretation challenging compared to logistic regression.
  • Generalizability to Other Populations: The models were developed and validated on a specific Brazilian cohort. Further validation on diverse populations is necessary to confirm their broader applicability.
  • Data Limitations: The study acknowledges that some participants were excluded due to missing data, which could potentially influence the results.
  • Future Work: The authors suggest refining the web tool for broader use in Brazil and exploring further improvements to the predictive models. They also highlight the ongoing evolution of machine learning techniques, implying potential for even more advanced models in the future.

Summary updated

Abstract

CONTEXT AND OBJECTIVE:: Type 2 diabetes is a chronic disease associated with a wide range of serious health complications that have a major impact on overall health. The aims here were to develop and validate predictive models for detecting undiagnosed diabetes using data from the Longitudinal Study of Adult Health (ELSA-Brasil) and to compare the performance of different machine-learning algorithms in this task. DESIGN AND SETTING:: Comparison of machine-learning algorithms to develop predictive models using data from ELSA-Brasil. METHODS:: After selecting a subset of 27 candidate variables from the literature, models were built and validated in four sequential steps: (i) parameter tuning with tenfold cross-validation, repeated three times; (ii) automatic variable selection using forward selection, a wrapper strategy with four different machine-learning algorithms and tenfold cross-validation (repeated three times), to evaluate each subset of variables; (iii) error estimation of model parameters with tenfold cross-validation, repeated ten times; and (iv) generalization testing on an independent dataset. The models were created with the following machine-learning algorithms: logistic regression, artificial neural network, naïve Bayes, K-nearest neighbor and random forest. RESULTS:: The best models were created using artificial neural networks and logistic regression. -These achieved mean areas under the curve of, respectively, 75.24% and 74.98% in the error estimation step and 74.17% and 74.41% in the generalization testing step. CONCLUSION:: Most of the predictive models produced similar results, and demonstrated the feasibility of identifying individuals with highest probability of having undiagnosed diabetes, through easily-obtained clinical data.
BibTeX
@article{olivera2017comparison,
  author = {André Rodrigues Olivera and Valter Roesler and Cirano Iochpe and María Inês Schmidt and Álvaro Vigo and Sandhi Maria Barreto and Bruce Bartholow Duncan},
  title = {{Comparison of machine-learning algorithms to build a predictive model for detecting undiagnosed diabetes - ELSA-Brasil: accuracy study}},
  journal = {Sao Paulo Medical Journal},
  year = {2017},
  doi = {10.1590/1516-3180.2016.0309010217},
}
Figure