Alzheimer's Disease Brain MRI Classification: Challenges and Insights

Yi Fung, Ziqiang Guan, Ritesh Kumar, Yeahuay Joie Wu, Madalina FiterauarXiv 2019 · OpenReview

Abstract

In recent years, many papers have reported state-of-the-art performance on Alzheimer's Disease classification with MRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset using convolutional neural networks. However, we discover that when we split that data into training and testing sets at the subject level, we are not able to obtain similar performance, bringing the validity of many of the previous studies into question. Furthermore, we point out that previous works use different subsets of the ADNI data, making comparison across similar works tricky. In this study, we present the results of three splitting methods, discuss the motivations behind their validity, and report our results using all of the available subjects.

Introduction

The application of deep learning to medical imaging, particularly for Alzheimer’s Disease (AD) diagnosis, has seen a surge in reported high-performance models. Most of these studies utilize the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. However, a significant discrepancy exists between reported accuracies and the generalizability of these models in clinical settings.

The core issue identified is improper data splitting, leading to "data leakage." Since the ADNI dataset often contains multiple longitudinal scans for a single patient, a random split at the image level results in the model seeing the same patient's brain structure in both the training and testing sets. In such cases, the model may simply memorize patient-specific anatomical features rather than learning generalizable biomarkers for AD.

Comparison of previous studies on AD classification using ADNI, highlighting varying data splitting approaches and high reported accuracies.

Methodology

To investigate the impact of data leakage, we implemented two primary architectures: a 2D ResNet18 (pre-trained on ImageNet) and a custom 22-layer 3D ResNet designed to capture volumetric spatial information.

The proposed 3D CNN architecture featuring residual blocks and convolutional layers.

We evaluated these models using three distinct data-splitting strategies:

  1. Random MRI Split: A naive approach where individual scans are shuffled and split, potentially placing scans from the same patient in both sets.
  2. Patient-Level Split: Scans are grouped by patient ID, ensuring that all scans from a single individual are restricted to either the training or testing set.
  3. Visit History Split: Models are trained on a patient’s earlier visits and tested on their final visit, simulating clinical progression prediction.

Dataset and Preprocessing

We utilized 2,731 MRI scans from 657 patients in the ADNI dataset, categorized into Cognitively Normal (CN), Mild Cognitive Impairment (MCI), and Alzheimer’s Disease (AD). All scans underwent spatial normalization using the Clinica T1-volume pipeline to ensure consistent orientation and scaling, a necessary step given the subtle anatomical differences between classes.

Results and Discussion

Our experiments reveal a dramatic performance drop when moving from image-level to patient-level splitting.

Model Split Method Training Acc. Testing Acc.
2D CNN Random MRI 99.2% 83.7%
2D CNN By Patient 98.8% 51.7%
3D CNN Random MRI 95.8% 84.4%
3D CNN By Patient 86.3% 52.4%

The results suggest that models achieving >80% accuracy on random splits are likely "memorizing" patients. When forced to generalize to unseen individuals, accuracy falls to near-chance levels (~52%). Furthermore, in the Visit History split, models struggled to detect actual disease transitions, often relying on the stability of the previous diagnosis.

Conclusion

This study highlights a critical need for rigorous evaluation standards in medical AI. High accuracy in AD classification can be an artifact of data leakage rather than true diagnostic capability. Future work should focus on multimodal learning—integrating genetic and cognitive data—and moving toward continuous cognitive score prediction to provide a richer learning signal.

Summary updated

Abstract

In recent years, many papers have reported state-of-the-art performance on Alzheimer's Disease classification with MRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset using convolutional neural networks. However, we discover that when we split that data into training and testing sets at the subject level, we are not able to obtain similar performance, bringing the validity of many of the previous studies into question. Furthermore, we point out that previous works use different subsets of the ADNI data, making comparison across similar works tricky. In this study, we present the results of three splitting methods, discuss the motivations behind their validity, and report our results using all of the available subjects.
BibTeX
@misc{fung2019alzheimer,
  author = {Yi Fung and Ziqiang Guan and Ritesh Kumar and Yeahuay Joie Wu and Madalina Fiterau},
  title = {{Alzheimer's Disease Brain MRI Classification: Challenges and Insights}},
  year = {2019},
  eprint = {1906.04231},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04231},
}
Figure