Cross-dataset Evaluation of Dementia Longitudinal Progression Prediction Models

Chen Zhang, Lijun An, Naren Wulan, Kim‐Ngan Nguyen, Csaba Orbán, Pansheng Chen, Christopher Chen, Juan Zhou, Keli Liu, B.t. Thomas Yeo, the Alzheimer’s Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle Study of AgingmedRxiv 2024 · DOI

Abstract: L2C-FNN shows strong potential for both short-term and long-term dementia progression prediction. Pretrained ADNI models are available: GitHub Repository.

Introduction

Predicting the longitudinal progression of Alzheimer’s disease (AD) is a critical challenge for clinical trial design and personalized patient care. While numerous algorithms have been developed to forecast cognitive decline and clinical diagnosis, their real-world utility is often limited by a "generalizability gap." Most models are trained and validated on single, highly controlled cohorts like the Alzheimer’s Disease Neuroimaging Initiative (ADNI), leaving their performance on diverse, global populations uncertain.

This study benchmarks top-performing progression models across four international datasets to evaluate their robustness and predictive power in heterogeneous clinical environments.

Methodology: The L2C-FNN Approach

The researchers focused on the "Longitudinal-to-Cross-sectional" (L2C) transformation. Unlike standard Recurrent Neural Networks (RNNs) that process data sequentially and often struggle with missing timepoints, the L2C approach "flattens" a patient's irregular history into a single, fixed-length feature vector.

The study introduces L2C-FNN, a novel variant that utilizes a Feedforward Neural Network for multi-task learning. Instead of training separate models for different outcomes, L2C-FNN simultaneously predicts three key indicators:

  1. Clinical Diagnosis (Cognitively Normal, MCI, or Dementia)
  2. Cognitive Performance (MMSE scores)
  3. Brain Atrophy (Ventricle volume)
Summary of baseline and recurring features used for model training and evaluation, categorized into demographic, genetic, MRI, and cognitive domains.

By learning shared representations across these tasks, the model achieves better regularization and improved accuracy compared to single-task models like the 2019 TADPOLE challenge winner, FROG.

Cross-Dataset Validation

To test generalizability, models were trained on the ADNI dataset (N=2111) and then evaluated on three external cohorts with distinct demographic and clinical profiles:

  • AIBL (Australia): A lifestyle-focused aging study.
  • MACC (Singapore): A cohort with significant vascular dementia representation.
  • OASIS (USA): An open-access series of imaging studies.
Demographic and clinical characteristics of the four datasets, highlighting significant differences in age, baseline MMSE, and diagnostic distribution.

These datasets vary significantly in disease severity and patient demographics, providing a rigorous "stress test" for the models' predictive capabilities.

Key Insights and Results

  • Superior Diagnostic Accuracy: L2C-FNN emerged as the most accurate model for clinical diagnosis prediction across all external datasets, outperforming both RNNs and parametric models.
  • Long-Term Robustness: While many models see a sharp decline in accuracy when predicting several years out, L2C-FNN maintained high performance for up to six years into the future. This stability is attributed to the L2C transformation, which avoids the error accumulation common in state-based sequential models.
  • Data Efficiency: The model remained competitive even when only one or two patient visits were available, suggesting its potential utility in early-stage clinical screening where longitudinal history is sparse.
  • Parametric vs. Data-Driven: While the parametric AD-Map model (which assumes biomarkers follow an S-shaped curve) performed well for continuous cognitive scores, it struggled with categorical diagnosis. L2C-FNN provided a more balanced and effective solution for both data types.

Limitations and Future Directions

Despite its success, the study identifies two primary areas for improvement. First, all models were trained exclusively on ADNI data; future research should explore multi-site training to better account for variations in MRI scanner types and population-specific factors. Second, the current comparison excluded high-impact biomarkers like PET scans and blood-based tau levels to maintain consistency across datasets. Integrating these biomarkers could further enhance the predictive power of the L2C-FNN framework.

Summary updated

Abstract

L2C-FNN shows strong potential for both short-term and long-term dementia progression prediction. Pretrained ADNI models are available: https://github.com/ThomasYeoLab/CBIG/tree/master/stable_projects/predict_phenotypes/Zhang2025_L2CFNN.
BibTeX
@misc{zhang2024cross,
  author = {Chen Zhang and Lijun An and Naren Wulan and Kim‐Ngan Nguyen and Csaba Orbán and Pansheng Chen and Christopher Chen and Juan Zhou and Keli Liu and B.T. Thomas Yeo and the Alzheimer’s Disease Neuroimaging Initiative and the Australian Imaging Biomarkers and Lifestyle Study of Aging},
  title = {{Cross-dataset Evaluation of Dementia Longitudinal Progression Prediction Models}},
  year = {2024},
  doi = {10.1101/2024.11.18.24317513},
  howpublished = {medRxiv},
}
Figure