Prediction of cognitive impairment via deep learning trained with multi-center neuropsychological test data

Min Ju Kang, Sang Yun Kim, Duk L. Na, Byeong C. Kim, Dong Won Yang, Eun-Joo Kim, Hae Ri Na, Hyun Jeong Han, Jae-Hong Lee, Jong Hun Kim, Kee Hyung Park, Kyung Won Park, Seol-Heui Han, Seong Yoon Kim, Soo Jin Yoon, Bora Yoon, Sang Won Seo, So Young Moon, Young-Soon Yang, Yong S. Shim, Min Jae Baek, Jee Hyang Jeong, Seong Hye Choi, Young Chul YounBMC Medical Informatics Decis. Mak 2019 · DOI

TL;DR

This paper introduces a deep learning model trained on a large, multi-center dataset of neuropsychological tests to predict cognitive impairment (CI), including mild cognitive impairment (MCI) and Alzheimer's disease dementia (ADD). The model achieves high accuracy, suggesting its potential as a valuable reference tool for clinicians in differentiating cognitive states.

What Problem Does This Paper Address?

Diagnosing cognitive impairment (CI) is crucial for timely intervention and management, but it relies heavily on neuropsychological tests (NPTs). Interpreting these tests requires specialized expertise and can be time-consuming. While NPTs are essential for identifying objective cognitive deficits, the process of analyzing them is often manual and dependent on the clinician's experience. This creates a bottleneck in clinical settings, especially when dealing with a large patient population. The paper aims to address this by developing an automated machine learning approach that can efficiently and accurately predict cognitive states from NPT data, thereby streamlining the diagnostic process and potentially reducing the burden on clinicians.

Methods

The researchers employed a deep learning approach using an artificial neural network (ANN) implemented with TensorFlow. They explored various ANN architectures, ultimately selecting a 3-layer network as the optimal model. The model was trained to classify subjects into three categories: normal cognition (NC), mild cognitive impairment (MCI), and Alzheimer's disease dementia (ADD). To enhance prediction accuracy and prevent overfitting, a dropout rate of 0.9 was utilized. Feature selection was performed using Recursive Feature Elimination (RFE) to identify the most influential NPT variables. The model was trained and evaluated using both balanced and imbalanced (clinic-based) datasets, with a 75/25 split for training and testing.

Datasets

The study utilized a substantial dataset comprising 14,926 formal neuropsychological assessments from the Seoul Neuropsychological Screening Battery (SNSB). This data was collected from a multi-center cohort study (CREDOS) involving three major hospitals: Seoul National University Bundang Hospital (BDSNUH), Chung-Ang University Hospital (CAUH), and the Clinical Research Center for Dementia of South Korea (CREDOS). The initial dataset contained 17,139 subjects, from which 2,213 were excluded due to missing diagnoses or severe white matter hyperintensities. The final dataset was then balanced to ensure an equal representation of diagnostic groups for certain analyses.

Key Insights and Ideas

  • Deep Learning's Efficacy in NPT Analysis: The study demonstrates that deep learning, specifically a 3-layer ANN, can effectively learn complex patterns within NPT data to predict cognitive impairment with high accuracy. This moves beyond traditional statistical methods and highlights the potential of AI in this domain.
  • Importance of Specific NPT Features: The RFE analysis revealed that "MMSE_orientation_to_time" and "MMSE_recall" were the most significant features for predicting cognitive impairment. This suggests that temporal orientation and short-term memory recall are particularly sensitive indicators.
  • Feature Reduction for Efficiency: The research shows that a reduced set of features (around 12-13 variables, including age and education) can achieve over 90% accuracy, indicating that a comprehensive set of 46 variables might not be strictly necessary for effective prediction, leading to more streamlined assessments.
  • Potential for Clinical Reference Tool: The high accuracy achieved, even with imbalanced datasets, suggests that this AI model could serve as a valuable reference tool for clinicians, aiding them in interpreting NPT results and potentially flagging cases that require closer attention.

Key Results

The deep learning model achieved impressive prediction accuracies:

  • 2-way classification (CI vs. NC):
    • Balanced dataset: 96.66 ± 0.52% accuracy.
  • Clinic-based dataset: 97.23 ± 0.32% accuracy.
  • 3-way classification (NC vs. MCI vs. ADD):
    • Balanced dataset: 95.49 ± 0.53% accuracy.
  • Clinic-based dataset: 96.34 ± 1.03% accuracy. The model also demonstrated high sensitivity and specificity, with values often exceeding 95%. The study found that the 3-layer neural network with a dropout rate of 0.9 performed optimally.

Limitations and Future Work

The authors acknowledge several limitations:

  • Inability to Predict Temporal Stage or Prognosis: The current model can classify cognitive states but cannot predict the progression of the disease or its future course.
  • Limited Scope of Dementia Subtypes: The "dementia" group primarily included Alzheimer's Disease Dementia (ADD), meaning the model may not be directly applicable to other subtypes of dementia.

Future work could focus on:

  • Developing algorithms that can predict disease progression and prognosis.
  • Expanding the model to include other dementia subtypes.
  • Further refining the algorithm for even higher classification efficiency and broader clinical utility.
  • Investigating the integration of other data modalities (e.g., MRI) to enhance diagnostic accuracy and address potential circularity issues.

Summary updated

BibTeX
@article{kang2019prediction,
  author = {Min Ju Kang and Sang Yun Kim and Duk L. Na and Byeong C. Kim and Dong Won Yang and Eun-Joo Kim and Hae Ri Na and Hyun Jeong Han and Jae-Hong Lee and Jong Hun Kim and Kee Hyung Park and Kyung Won Park and Seol-Heui Han and Seong Yoon Kim and Soo Jin Yoon and Bora Yoon and Sang Won Seo and So Young Moon and Young-Soon Yang and Yong S. Shim and Min Jae Baek and Jee Hyang Jeong and Seong Hye Choi and Young Chul Youn},
  title = {{Prediction of cognitive impairment via deep learning trained with multi-center neuropsychological test data}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2019},
  doi = {10.1186/s12911-019-0974-x},
}
Figure