The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up

Razvan Marinescu, Neil P. Oxtoby, Alexandra L. Young, Esther E. Bron, Arthur W. Toga, Michael W. Weiner, Frederik Barkhof, Nick C. Fox, Arman Eshaghi, Tina Toni, Marcin Salaterski, Veronika Lunina, Manon Ansart, Stanley Durrleman, Pascal Lu, Samuel Iddi, Dan Li, Wesley K. Thompson, Michael Donohue, Aviv Nahon, Yarden Levy, Dan Halbersberg, Mariya Cohen, Huiling Liao, Tengfei Li, Kaixian Yu, Hongtu Zhu, José G. Tamez‐Peña, Aya Ismail, Timothy C. Wood, Héctor Corrada Bravo, Minh Nguyen, Nanbo Sun, Jiashi Feng, B.t. Thomas Yeo, Gang Chen, Ke Qi, Shiyang Chen, Deqiang Qiu, Ionut Buciuman, Alex Kelner, Raluca Maria Pop, Denisa Rimocea, Mostafa Mehdipour Ghazi, Mads Nielsen, Sébastien Ourselin, Lauge Sørensen, Vikram Venkatraghavan, Keli Liu, Christina Rabe, Paul T. Manser, Steven M. Hill, James Howlett, Zhiyue Huang, Steven J. Kiddle, Sach Mukherjee, Anaïs Rouanet, Bernd Taschler, Brian D. M. Tom, Simon R. White, Noel G. Faux, Suman Sedai, Javier de Velasco Oriol, Edgar E. V. Clemente, Karol Estrada, Leon Aksman, André Altmann, Cynthia M. Stonnington, Yalin Wang, Jianfeng Wu, Vivek Devadas, Clémentine Fourrier, Lars Lau Rakêt, Aristeidis Sotiras, Güray Erus, Jimit Doshi, Christos Davatzikos, Jacob W. Vogel, Andrew Doyle, Angela Tam, Alex Diaz-Papkovich, Emmanuel Jammeh, Igor Koval, Paul J. Moore, Terry Lyons, John Gallacher, Jussi Tohka, Robert Ciszek, Bruno Jedynak, K. Pandya, Murat Bilgel, William R. Engels, Joseph B. Cole, Polina Golland, Stefan Klein, Daniel C. Alexander, The Alzheimer's Disease Neuroimaging InitiativeThe Journal of Machine Learning for Biomedical Imaging 2021 · DOI

Abstract

Accurate prediction of progression in subjects at risk of Alzheimer's disease is crucial for enrolling the right subjects in clinical trials. However, a prospective comparison of state-of-the-art algorithms for predicting disease onset and progression is currently lacking. We present the findings of "The Alzheimer's Disease Prediction Of Longitudinal Evolution" (TADPOLE) Challenge, which compared the performance of 92 algorithms from 33 international teams at predicting the future trajectory of 219 individuals at risk of Alzheimer's disease. Challenge participants were required to make a prediction, for each month of a 5-year future time period, of three key outcomes: clinical diagnosis, Alzheimer's Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13), and total volume of the ventricles. The methods used by challenge participants included multivariate linear regression, machine learning methods such as support vector machines and deep neural networks, as well as disease progression models. No single submission was best at predicting all three outcomes. For clinical diagnosis and ventricle volume prediction, the best algorithms strongly outperform simple baselines in predictive ability. However, for ADAS-Cog13 no single submitted prediction method was significantly better than random guesswork. Two ensemble methods based on taking the mean and median over all predictions, obtained top scores on almost all tasks. Better than average performance at diagnosis prediction was generally associated with the additional inclusion of features from cerebrospinal fluid (CSF) samples and diffusion tensor imaging (DTI). On the other hand, better performance at ventricle volume prediction was associated with inclusion of summary statistics, such as the slope or maxima/minima of patient-specific biomarkers. On a limited, cross-sectional subset of the data emulating clinical trials, performance of the best algorithms at predicting clinical diagnosis decreased only slightly (2 percentage points) compared to the full longitudinal dataset. The submission system remains open via the website https://tadpole.grand-challenge.org, while TADPOLE SHARE (https://tadpole-share.github.io/) collates code for submissions. TADPOLE's unique results suggest that current prediction algorithms provide sufficient accuracy to exploit biomarkers related to clinical diagnosis and ventricle volume, for cohort refinement in clinical trials for Alzheimer's disease. However, results call into question the usage of cognitive test scores for patient selection and as a primary endpoint in clinical trials.

Introduction

Predicting how an individual’s Alzheimer’s Disease (AD) will evolve is a primary goal for clinical research. To test new drugs effectively, researchers must enroll participants who are at high risk of progressing during the trial period. However, the field has been historically plagued by "publication bias" and inconsistent benchmarking. Many studies claim high accuracy but utilize different datasets, processing pipelines, and evaluation metrics. Furthermore, most are retrospective—they "predict" the past. The TADPOLE (The Alzheimer’s Disease Prediction Of Longitudinal Evolution) Challenge addressed this by requiring participants to submit forecasts for data that did not yet exist, creating a true prospective benchmark for the state-of-the-art in AD modeling.

Methods and Modeling Approaches

The challenge invited 33 international teams to submit 92 different algorithms. These methods spanned a wide spectrum of computational approaches, including statistical regression (multivariate linear and mixed-effects models), machine learning (Support Vector Machines, Random Forests, Gradient Boosting), and deep learning (RNNs and LSTMs). Specialized Disease Progression Models (DPMs) were also employed to reconstruct disease timelines without relying on fixed clinical labels.

Table summarizing the diverse modeling approaches, feature selection strategies, and computational requirements for a subset of the challenge submissions.

A standout methodological finding was the power of ensemble methods. Simple "consensus" models—taking the mean or median of all 92 submissions—consistently outperformed almost every individual algorithm, suggesting that different algorithms make complementary errors that cancel out when combined.

Datasets

The primary data source was the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The organizers curated four specific subsets to facilitate training and evaluation:

  • D1 & D2: Comprehensive longitudinal datasets used for training.
  • D3: A "cross-sectional" set containing only the most recent visit for each patient, designed to simulate the limited data typically available during clinical trial screening.
  • D4 (The Test Set): Data from 219 individuals collected after the submission deadline, ensuring a truly prospective evaluation.
Demographic and clinical characteristics of subjects across the four TADPOLE datasets, showing the distribution of Controls, MCI, and AD subjects.

Key Results and Insights

The challenge revealed a stark contrast in our ability to predict different aspects of AD progression:

  • The "Cognitive Wall": The most surprising finding was that ADAS-Cog13 (a standard cognitive test) is extremely difficult to forecast. No individual method was significantly better than random guesswork. This suggests cognitive scores may be too noisy or influenced by external factors like "practice effects" to serve as reliable short-term predictors.
  • Predictable Atrophy: In contrast, brain atrophy (ventricle volume) is highly predictable. The best models achieved a Mean Absolute Error (MAE) of 0.41% of intracranial volume. Success here was often associated with using "augmented features" like the historical slope of change.
  • Diagnostic Accuracy: The best models achieved a Multiclass Area Under the Curve (MAUC) of 0.931 (Team Frog). Performance was generally boosted by including Cerebrospinal Fluid (CSF) biomarkers and Diffusion Tensor Imaging (DTI).
  • Cross-sectional Utility: Models trained on the limited D3 set performed nearly as well at predicting diagnosis as those trained on full longitudinal histories. This is encouraging for the feasibility of real-world clinical trial recruitment where long-term history is often unavailable.

Discussion and Limitations

While the results suggest that current algorithms provide sufficient accuracy for cohort refinement (choosing the right people for a trial), they call into question the usage of cognitive test scores as primary endpoints. Limitations of the study include a relatively small test sample (219 subjects) and a short one-year follow-up period. Alzheimer’s is a slow-progressing disease, and the organizers plan to re-evaluate these same 92 algorithms after 3–5 years of data collection to assess long-term predictive stability.

Summary updated

Abstract

Accurate prediction of progression in subjects at risk of Alzheimer's disease is crucial for enrolling the right subjects in clinical trials. However, a prospective comparison of state-of-the-art algorithms for predicting disease onset and progression is currently lacking. We present the findings of "The Alzheimer's Disease Prediction Of Longitudinal Evolution" (TADPOLE) Challenge, which compared the performance of 92 algorithms from 33 international teams at predicting the future trajectory of 219 individuals at risk of Alzheimer's disease. Challenge participants were required to make a prediction, for each month of a 5-year future time period, of three key outcomes: clinical diagnosis, Alzheimer's Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13), and total volume of the ventricles. The methods used by challenge participants included multivariate linear regression, machine learning methods such as support vector machines and deep neural networks, as well as disease progression models. No single submission was best at predicting all three outcomes. For clinical diagnosis and ventricle volume prediction, the best algorithms strongly outperform simple baselines in predictive ability. However, for ADAS-Cog13 no single submitted prediction method was significantly better than random guesswork. Two ensemble methods based on taking the mean and median over all predictions, obtained top scores on almost all tasks. Better than average performance at diagnosis prediction was generally associated with the additional inclusion of features from cerebrospinal fluid (CSF) samples and diffusion tensor imaging (DTI). On the other hand, better performance at ventricle volume prediction was associated with inclusion of summary statistics, such as the slope or maxima/minima of patient-specific biomarkers. On a limited, cross-sectional subset of the data emulating clinical trials, performance of the best algorithms at predicting clinical diagnosis decreased only slightly (2 percentage points) compared to the full longitudinal dataset. The submission system remains open via the website https://tadpole.grand-challenge.org, while TADPOLE SHARE (https://tadpole-share.github.io/) collates code for submissions. TADPOLE's unique results suggest that current prediction algorithms provide sufficient accuracy to exploit biomarkers related to clinical diagnosis and ventricle volume, for cohort refinement in clinical trials for Alzheimer's disease. However, results call into question the usage of cognitive test scores for patient selection and as a primary endpoint in clinical trials.
BibTeX
@article{marinescu2021alzheimer,
  author = {Razvan Marinescu and Neil P. Oxtoby and Alexandra L. Young and Esther E. Bron and Arthur W. Toga and Michael W. Weiner and Frederik Barkhof and Nick C. Fox and Arman Eshaghi and Tina Toni and Marcin Salaterski and Veronika Lunina and Manon Ansart and Stanley Durrleman and Pascal Lu and Samuel Iddi and Dan Li and Wesley K. Thompson and Michael Donohue and Aviv Nahon and Yarden Levy and Dan Halbersberg and Mariya Cohen and Huiling Liao and Tengfei Li and Kaixian Yu and Hongtu Zhu and José G. Tamez‐Peña and Aya Ismail and Timothy C. Wood and Héctor Corrada Bravo and Minh Nguyen and Nanbo Sun and Jiashi Feng and B.T. Thomas Yeo and Gang Chen and Ke Qi and Shiyang Chen and Deqiang Qiu and Ionut Buciuman and Alex Kelner and Raluca Maria Pop and Denisa Rimocea and Mostafa Mehdipour Ghazi and Mads Nielsen and Sébastien Ourselin and Lauge Sørensen and Vikram Venkatraghavan and Keli Liu and Christina Rabe and Paul T. Manser and Steven M. Hill and James Howlett and Zhiyue Huang and Steven J. Kiddle and Sach Mukherjee and Anaïs Rouanet and Bernd Taschler and Brian D. M. Tom and Simon R. White and Noel G. Faux and Suman Sedai and Javier de Velasco Oriol and Edgar E. V. Clemente and Karol Estrada and Leon Aksman and André Altmann and Cynthia M. Stonnington and Yalin Wang and Jianfeng Wu and Vivek Devadas and Clémentine Fourrier and Lars Lau Rakêt and Aristeidis Sotiras and Güray Erus and Jimit Doshi and Christos Davatzikos and Jacob W. Vogel and Andrew Doyle and Angela Tam and Alex Diaz-Papkovich and Emmanuel Jammeh and Igor Koval and Paul J. Moore and Terry Lyons and John Gallacher and Jussi Tohka and Robert Ciszek and Bruno Jedynak and K. Pandya and Murat Bilgel and William R. Engels and Joseph B. Cole and Polina Golland and Stefan Klein and Daniel C. Alexander and The Alzheimer's Disease Neuroimaging Initiative},
  title = {{The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up}},
  journal = {The Journal of Machine Learning for Biomedical Imaging},
  year = {2021},
  doi = {10.59275/j.melba.2021-2dcc},
}
Figure