Multimodal deep learning for Alzheimer’s disease dementia assessment

Shangran Qiu, Matthew I. Miller, Prajakta Joshi, Joyce C. Lee, Chonghua Xue, Yunruo Ni, Yuwei Wang, Ileana De Anda‐Duran, Phillip H Hwang, Justin Cramer, Brigid Dwyer, Honglin Hao, Michelle Kaku, Sachin Kedar, Peter H. Lee, Asim Mian, Daniel L. Murman, Sarah A. O’Shea, Aaron B. Paul, Marie Saint‐Hilaire, E. Alton Sartor, Aneeta Saxena, Ludy C. Shih, Juan E. Small, Maximilian J. Smith, Arun Swaminathan, Courtney Takahashi, Olga Taraschenko, Hui You, Jing Yuan, Yan Zhou, Shuhan Zhu, Michael L. Alosco, Jesse Mez, Thor D. Stein, Kathleen L. Poston, Rhoda Au, Vijaya B. KolachalamaNature Communications 2022 · DOI

TL;DR

This paper introduces a multimodal deep learning framework that accurately classifies individuals into normal cognition (NC), mild cognitive impairment (MCI), Alzheimer's disease (AD), and non-AD dementias (nADD) using a flexible combination of clinical data. The models achieve diagnostic accuracy comparable to practicing neurologists and neuroradiologists, and their predictions are validated against neuropathological findings.

What Problem Does This Paper Address?

Alzheimer's disease (AD) is a leading cause of dementia, with millions of new cases annually. Early and accurate diagnosis is crucial for effective management and treatment, but current diagnostic methods often rely on complex clinical assessments, neuropsychological testing, and neuroimaging, which can be time-consuming and require specialized expertise. Furthermore, differentiating AD from other forms of dementia (nADD) can be challenging due to overlapping symptoms. This paper aims to develop a computational framework that can assist in this complex differential diagnosis by integrating various data sources and achieving expert-level accuracy.

Methods

The authors developed a deep learning framework that operates in a two-stage process:

  1. Cognitive Status Prediction (COG task): This stage classifies individuals into Normal Cognition (NC), Mild Cognitive Impairment (MCI), or Dementia (DE). They introduced a "Dementia MOdel (DEMO)" score, a continuous value from 0 to 2, to represent cognitive status.
  2. Dementia Etiology Prediction (ADD task): For individuals classified as DE, this stage further differentiates between Alzheimer's Disease (AD) and non-Alzheimer's Disease Dementias (nADD). They developed an "ALZheimer (ALZ)" score to represent the probability of AD.

They explored three model architectures:

  • MRI-only model: A convolutional neural network (CNN) that processes MRI scans to generate DEMO and ALZ scores.
  • Non-imaging model: A traditional machine learning classifier (specifically CatBoost) that uses demographic, medical history, neuropsychological, and functional assessment data.
  • Fusion model: A hybrid approach that combines the CNN for MRI processing with a CatBoost classifier, integrating both imaging and non-imaging data.

To ensure interpretability and validate their findings, they employed SHAP (SHapley Additive exPlanations) values to understand which features contribute most to the model's predictions and to map these contributions to specific brain regions.

Datasets

The study utilized data from eight independent cohorts:

  • National Alzheimer's Coordinating Center (NACC) - Used for model development and internal validation.
  • Alzheimer's Disease Neuroimaging Initiative (ADNI)
  • Australian Imaging, Biomarker and Lifestyle (AIBL)
  • Framingham Heart Study (FHS)
  • Lewy Body Dementia Center for Excellence at Stanford University (LBDSU)
  • Neuroimaging Initiative for Frontotemporal Lobar Degeneration (NIFD)
  • Open Access Series of Imaging Studies-3 (OASIS) - Used for external validation.
  • Parkinson's Progression Marker Initiative (PPMI)

Key Insights and Ideas

  • Multimodal data is key: Combining imaging (MRI) with clinical data (demographics, history, neuropsychological tests) significantly enhances diagnostic accuracy compared to using either modality alone.
  • Interpretable AI for healthcare: The use of SHAP values allows for the visualization of which brain regions and clinical features are driving the diagnostic predictions, fostering trust and understanding in AI-driven medical tools.
  • Bridging the gap to clinical practice: The framework is designed to be flexible with input data, making it adaptable to different clinical settings, from general practitioners to specialized memory clinics.
  • Validation against gold standards: The models' predictions were not only compared to expert clinicians but also validated against neuropathological findings from autopsies, providing a robust measure of their clinical relevance.

Key Results

  • The fusion model achieved the highest diagnostic accuracy across various tasks, including differentiating between NC, MCI, AD, and nADD.
  • For the 4-way classification (NC, MCI, AD, nADD), the fusion model achieved an accuracy of 0.777 ± 0.011 on the NACC test set and 0.730 ± 0.045 on the OASIS external validation set.
  • The models demonstrated performance comparable to practicing neurologists in cognitive status assessment and neuroradiologists in distinguishing AD from nADD.
  • SHAP analysis revealed that the DEMO score (derived from CNNs) and MMSE score were crucial for cognitive status prediction, while the ALZ score (from CNNs) was paramount for AD vs. nADD classification.
  • Neuroimaging signatures identified by SHAP analysis showed distinct patterns of atrophy in brain regions associated with AD and nADD, correlating with neuropathological findings.

Limitations and Future Work

  • Mixed dementia cases: The current models default to an AD diagnosis when mixed dementia is present. Future work could involve multi-label classification to identify co-occurring conditions.
  • Atypical AD: The study did not include confirmed cases of atypical AD, which could be a focus for future research.
  • MCI subtypes: The study did not differentiate between amnestic and non-amnestic MCI. Future work could explore distinct signatures for these subtypes.
  • Generalizability to rare dementias: While the models perform well, there's a possibility of decreased accuracy for less common dementias. Future efforts could incorporate specific clinical data tailored to these conditions.
  • Comparison with consensus reviews: Future studies could compare the model's performance against teams of collaborating clinicians rather than individual experts.

The authors envision a prospective observational study in memory clinics to further validate their model's performance against expert clinicians and explore its potential as an assistive screening tool.

Summary updated

Abstract

Worldwide, there are nearly 10 million new cases of dementia annually, of which Alzheimer's disease (AD) is the most common. New measures are needed to improve the diagnosis of individuals with cognitive impairment due to various etiologies. Here, we report a deep learning framework that accomplishes multiple diagnostic steps in successive fashion to identify persons with normal cognition (NC), mild cognitive impairment (MCI), AD, and non-AD dementias (nADD). We demonstrate a range of models capable of accepting flexible combinations of routinely collected clinical information, including demographics, medical history, neuropsychological testing, neuroimaging, and functional assessments. We then show that these frameworks compare favorably with the diagnostic accuracy of practicing neurologists and neuroradiologists. Lastly, we apply interpretability methods in computer vision to show that disease-specific patterns detected by our models track distinct patterns of degenerative changes throughout the brain and correspond closely with the presence of neuropathological lesions on autopsy. Our work demonstrates methodologies for validating computational predictions with established standards of medical diagnosis.
BibTeX
@inproceedings{qiu2022multimodal,
  author = {Shangran Qiu and Matthew I. Miller and Prajakta Joshi and Joyce C. Lee and Chonghua Xue and Yunruo Ni and Yuwei Wang and Ileana De Anda‐Duran and Phillip H Hwang and Justin Cramer and Brigid Dwyer and Honglin Hao and Michelle Kaku and Sachin Kedar and Peter H. Lee and Asim Mian and Daniel L. Murman and Sarah A. O’Shea and Aaron B. Paul and Marie Saint‐Hilaire and E. Alton Sartor and Aneeta Saxena and Ludy C. Shih and Juan E. Small and Maximilian J. Smith and Arun Swaminathan and Courtney Takahashi and Olga Taraschenko and Hui You and Jing Yuan and Yan Zhou and Shuhan Zhu and Michael L. Alosco and Jesse Mez and Thor D. Stein and Kathleen L. Poston and Rhoda Au and Vijaya B. Kolachalama},
  title = {{Multimodal deep learning for Alzheimer’s disease dementia assessment}},
  booktitle = {Nature Communications},
  year = {2022},
  doi = {10.1038/s41467-022-31037-5},
}
Figure