Random Forest Model in the Diagnosis of Dementia Patients with Normal Mini-Mental State Examination Scores

Jie Wang, Zhuo Wang, Ning Liu, Caiyan Liu, Chenhui Mao, Liling Dong, Jie Li, Xinying Huang, Dan Lei, Shanshan Chu, Jianyong Wang, Jing GaoJournal of Personalized Medicine 2022 · DOI

TL;DR

This paper introduces a Random Forest (RF) model that effectively diagnoses Mild Cognitive Impairment (MCI) and dementia in individuals with normal Mini-Mental State Examination (MMSE) scores. By optimizing a neuropsychological test battery (NTB) using machine learning, they developed a more efficient and accurate diagnostic tool, reducing evaluation time by half.

What Problem Does This Paper Address?

The Mini-Mental State Examination (MMSE) is a widely used, quick screening tool for cognitive impairment. However, it has limitations, particularly in detecting early-stage cognitive decline. A significant number of individuals with normal MMSE scores (typically >26) can still have underlying cognitive impairments, including Mild Cognitive Impairment (MCI) and even dementia. This "missed diagnosis" can delay crucial interventions. To address this, a more comprehensive neuropsychological assessment is needed, but existing batteries can be time-consuming. This paper aims to create a more efficient and accurate diagnostic tool for these individuals by leveraging machine learning to optimize the selection of neuropsychological tests.

Methods

The researchers employed a machine learning approach, specifically focusing on the Random Forest (RF) algorithm, to identify the most informative neuropsychological tests for diagnosing MCI and dementia in individuals with MMSE scores above 26. They began by extracting 64 features, encompassing demographic information and scores from a comprehensive neuropsychological test battery (NTB).

The core of their method involved:

  • Feature Importance Analysis: Using RF, they ranked the importance of each of the 64 features in distinguishing between cognitively unimpaired (CU), MCI, and dementia groups.
  • Feature Selection: Based on this importance ranking, they selected a subset of the most predictive features. This process aimed to simplify the NTB and reduce evaluation time without sacrificing diagnostic accuracy.
  • Model Comparison: They trained and compared the performance of several machine learning algorithms, including logistic regression, decision tree, SVM, XGBoost, and RF, using the selected features.
  • Model Evaluation: Performance was evaluated using metrics such as accuracy, precision, recall, F1 score, and Area Under the Receiver Operating Characteristic Curve (AUC).

Datasets

The study utilized data from 375 participants enrolled in the Peking Union Medical College Hospital (PUMCH) dementia cohort. These participants were categorized into three groups:

  • Cognitively Unimpaired (CU): n = 67
  • Mild Cognitive Impairment (MCI): n = 174
  • Dementia: n = 134

All participants had an MMSE score greater than 26. The data included demographic information and scores from a comprehensive neuropsychological test battery (NTB) designed for the Chinese population.

Key Insights and Ideas

  • MMSE is Insufficient for Early Detection: The study strongly reinforces that an MMSE score of 26 or higher does not guarantee the absence of cognitive impairment, highlighting the need for more sensitive diagnostic tools.
  • Machine Learning for NTB Optimization: RF proved to be a powerful tool for identifying which specific neuropsychological tests and their sub-components are most crucial for differentiating between cognitive states, allowing for a streamlined assessment.
  • Efficiency Gains are Significant: By reducing the number of tests from a comprehensive battery to a select few, the evaluation time was effectively halved, making cognitive assessments more practical.
  • Episodic Memory is Key, but Not Exclusive: While tests related to episodic memory (like AVLT-H and LMT) consistently emerged as important, the optimized model also incorporated elements from executive function and visuospatial abilities, suggesting a multi-domain approach is superior.

Key Results

  • Random Forest Dominance: The Random Forest (RF) classifier outperformed other tested models (logistic regression, decision tree, SVM, XGBoost) in identifying MCI and dementia.
  • Optimized NTB Performance: The final optimized NTB, consisting of six neuropsychological subtests and 22 high-importance features, achieved an AUC of 0.89 for distinguishing MCI from CU and 0.84 for distinguishing dementia from non-dementia.
  • Reduced Test Time: The optimized battery, which includes tests like AVLT-H, PAL, modified Rey figure, LMT, DST, and TMT A, reduced the evaluation time from over an hour to approximately 30 minutes.
  • High Sensitivity and Specificity: The RF model demonstrated good sensitivity (0.87) and specificity (0.85) for differentiating MCI from CU, and good sensitivity (0.85) and specificity (0.73) for dementia from non-dementia.

Limitations and Future Work

  • Retrospective and Single-Center Design: The study acknowledges its retrospective nature and single-center origin, which may introduce selection bias. Future work should involve prospective, multi-centered studies to validate these findings.
  • Dementia Subtyping: The study did not differentiate between various subtypes of dementia (e.g., Alzheimer's disease, vascular dementia). This lack of specificity might have influenced the importance of certain features, particularly those related to language. Future research should explore how the model performs with specific dementia subtypes.
  • Normative Data Updates: The paper notes that the normative data for the NTB used in the study may not be fully up-to-date due to improvements in education and living conditions in China. Re-establishing norms for large samples is a time-consuming task, and ML offers a way to adapt to these changes.
  • Generalizability: While the NTB is tailored for the Chinese population, further validation in diverse cultural and linguistic contexts would be beneficial.

Summary updated

Abstract

<b>Background:</b> Mini-Mental State Examination (MMSE) is the most widely used tool in cognitive screening. Some individuals with normal MMSE scores have extensive cognitive impairment. Systematic neuropsychological assessment should be performed in these patients. This study aimed to optimize the systematic neuropsychological test battery (NTB) by machine learning and develop new classification models for distinguishing mild cognitive impairment (MCI) and dementia among individuals with MMSE ≥ 26. <b>Methods:</b> 375 participants with MMSE ≥ 26 were assigned a diagnosis of cognitively unimpaired (CU) (<i>n</i> = 67), MCI (<i>n</i> = 174), or dementia (<i>n</i> = 134). We compared the performance of five machine learning algorithms, including logistic regression, decision tree, SVM, XGBoost, and random forest (RF), in identifying MCI and dementia. <b>Results:</b> RF performed best in identifying MCI and dementia. Six neuropsychological subtests with high-importance features were selected to form a simplified NTB, and the test time was cut in half. The AUC of the RF model was 0.89 for distinguishing MCI from CU, and 0.84 for distinguishing dementia from nondementia. <b>Conclusions:</b> This simplified cognitive assessment model can be useful for the diagnosis of MCI and dementia in patients with normal MMSE. It not only optimizes the content of cognitive evaluation, but also improves diagnosis and reduces missed diagnosis.
BibTeX
@article{wang2022random,
  author = {Jie Wang and Zhuo Wang and Ning Liu and Caiyan Liu and Chenhui Mao and Liling Dong and Jie Li and Xinying Huang and Dan Lei and Shanshan Chu and Jianyong Wang and Jing Gao and Jianyong Wang and Jing Gao},
  title = {{Random Forest Model in the Diagnosis of Dementia Patients with Normal Mini-Mental State Examination Scores}},
  journal = {Journal of Personalized Medicine},
  year = {2022},
  doi = {10.3390/jpm12010037},
}
Figure