Analyzing large Alzheimer's disease cognitive datasets: Considerations and challenges

Maura Bellio, Neil P. Oxtoby, Zuzana Walker, Susie M.d. Henley, Annemie Ribbens, Ann Blandford, Daniel C. Alexander, Keir YongAlzheimer 2020 · DOI

TL;DR

As Alzheimer's disease (AD) research increasingly leverages large, shared datasets, computational researchers face challenges in interpreting complex cognitive test data. This paper provides a crucial guide for non-specialists, outlining the nuances of cognitive tests, common pitfalls in their analysis, and best practices to ensure valid insights into AD progression and heterogeneity.

What Problem Does This Paper Address?

The explosion of data-sharing initiatives in Alzheimer's disease (AD) research has opened doors for computational approaches to analyze vast amounts of clinical and preclinical data. A cornerstone of these datasets are cognitive assessments, which are vital for diagnosing AD, tracking its progression, and defining patient phenotypes. However, computational researchers, often lacking deep clinical expertise, can struggle with the inherent complexity and multifactorial nature of cognitive tests. These tests are not always straightforward measures of a single cognitive domain; they can be influenced by practice effects, ceiling/floor effects, and individual differences. Misinterpreting this data can lead to flawed conclusions about AD progression and heterogeneity. This paper aims to bridge this gap by demystifying cognitive test data for computational scientists, highlighting potential pitfalls, and offering guidance for more accurate and insightful analysis.

Methods

This paper is a perspective piece, not a traditional empirical study. Therefore, it doesn't present novel algorithms or computational methods. Instead, it focuses on providing a framework and guidance for the application of existing data-driven methods to cognitive test data. The authors outline a flowchart (Figure 1) that guides researchers through the process:

  • Defining the Research Question: Emphasizing the need for clear questions that align with the capabilities of cognitive tests.
  • Test Selection: Discussing how to choose appropriate tests based on the research question, considering measures for screening, diagnosis, progression, and discrete cognitive functions.
  • Quality Control & Standardization: Highlighting critical steps like evaluating data quality, accounting for scoring heterogeneity, and addressing characteristic test effects (practice, floor/ceiling).
  • Models and Assumptions: Briefly touching upon the selection of appropriate statistical and computational models and their underlying assumptions.

The paper's "method" is essentially a comprehensive review and synthesis of best practices for handling and analyzing cognitive test data within the context of AD research, drawing on existing literature and common practices in the field.

Datasets

The paper discusses several prominent AD data collection initiatives that provide access to large, multimodal datasets. These include:

  • ADNI (Alzheimer's Disease Neuroimaging Initiative): A major longitudinal study collecting clinical, neuroimaging, biospecimen, and cognitive data.
  • LAADC (Layton Aging & Alzheimer's Disease Center): Focuses on preclinical and early dementia, collecting clinical, MRI, and genetic data.
  • NACC (National Alzheimer's Coordinating Center): A large repository of clinical evaluations and neuropathology data.

The paper notes that these initiatives often include a wide array of cognitive tests, and it provides examples of commonly featured tests (e.g., MMSE, MoCA, ADAS-cog, CFT, TMT A-B) in Table 1. The authors do not introduce new datasets or benchmarks but rather discuss how to effectively utilize the data from these existing, publicly available resources.

Key Insights and Ideas

  • Cognitive Tests are Multifaceted: A critical insight is that cognitive tests are rarely pure measures of a single domain. They are influenced by a complex interplay of factors, including attention, executive function, and even visuospatial abilities, making direct interpretation of a single score challenging.
  • The "Black Box" Problem: Computational researchers often approach data with a "black box" mentality. However, with cognitive data, understanding the "mechanics" of the tests (their scoring, what they measure, their limitations) is crucial for avoiding misinterpretations.
  • Practice Effects are Pervasive: Repeated administration of cognitive tests, common in longitudinal studies, leads to practice effects (improvement due to familiarity). This can artificially inflate scores and mask true cognitive decline if not properly accounted for.
  • Floor and Ceiling Effects Limit Measurement: Many tests have limitations at the extremes of performance. Early-stage AD patients might perform perfectly (ceiling effect), while severely impaired patients might score zero (floor effect), making it difficult to capture subtle changes or differentiate performance.
  • Heterogeneity is Key, but Tricky: AD presents with significant phenotypic heterogeneity. While data-driven methods can help identify subtypes, understanding how cognitive test characteristics might align with or obscure these subtypes is vital.
  • Digitalization is the Future: The paper highlights the growing importance of digital cognitive tests, which offer standardized scoring, detailed reaction time data, and potential for adaptive testing to mitigate floor/ceiling effects, paving the way for more robust and scalable assessments.

Key Results

As a perspective paper, it doesn't present quantitative results in the traditional sense. Instead, its "results" are the insights and recommendations derived from its analysis of cognitive test data. The key takeaways are:

  • Identification of Common Pitfalls: The paper details issues like scoring heterogeneity, inappropriate use of normative data, practice effects, and floor/ceiling effects, providing concrete examples of how these can lead to misinterpretation.
  • Guidance for Test Selection: It offers a structured approach to selecting cognitive tests that are appropriate for specific research questions, distinguishing between measures for screening, diagnosis, and tracking progression.
  • Emphasis on Context: The overarching finding is that cognitive test scores must always be interpreted within the broader clinical context of the individual and the specific characteristics of the tests used.

Limitations and Future Work

Limitations:

  • Scope: The paper acknowledges that providing an exhaustive summary of all factors influencing cognitive test data is beyond its scope.
  • Generalizability: While discussing common tests, the paper notes that the specific battery of tests used can vary significantly across clinical contexts and research initiatives.
  • "Black Box" Concern: The paper touches upon the broader criticism of "black box" data-driven models in healthcare, noting that interpretability remains a challenge, especially when clinical decisions are influenced by these models.

Future Work:

  • Digital Cognitive Assessments: The paper strongly advocates for the increased adoption and validation of digital cognitive tests, which offer more granular data and potential for adaptive testing.
  • Eye-Tracking and Behavioral Markers: Future research could leverage technologies like eye-tracking and other non-task-engagement behavioral markers to identify subtle biomarkers for early AD detection and progression.
  • Improved Interpretability of Models: There's a need for continued efforts to improve the interpretability of machine learning models used in AD research, ensuring that findings are not only statistically significant but also clinically meaningful.
  • Cross-Disciplinary Collaboration: The paper emphasizes the ongoing need for collaboration between computational scientists and clinicians to ensure that data-driven approaches are clinically relevant and that findings are accurately interpreted.

Summary updated

Abstract

Recent data-sharing initiatives of clinical and preclinical Alzheimer's disease (AD) have led to a growing number of non-clinical researchers analyzing these datasets using modern data-driven computational methods. Cognitive tests are key components of such datasets, representing the principal clinical tool to establish phenotypes and monitor symptomatic progression. Despite the potential of computational analyses in complementing the clinical understanding of AD, the characteristics and multifactorial nature of cognitive tests are often unfamiliar to computational researchers and other non-specialist audiences. This perspective paper outlines core features, idiosyncrasies, and applications of cognitive test data. We report tests commonly featured in data-sharing initiatives, highlight key considerations in their selection and analysis, and provide suggestions to avoid risks of misinterpretation. Ultimately, the greater transparency of cognitive measures will maximize insights offered in AD, particularly regarding understanding the extent and basis of AD phenotypic heterogeneity.
BibTeX
@inproceedings{bellio2020analyzing,
  author = {Maura Bellio and Neil P. Oxtoby and Zuzana Walker and Susie M.D. Henley and Annemie Ribbens and Ann Blandford and Daniel C. Alexander and Keir Yong},
  title = {{Analyzing large Alzheimer's disease cognitive datasets: Considerations and challenges}},
  booktitle = {Alzheimer},
  year = {2020},
  doi = {10.1002/dad2.12135},
}
Figure