Investigating test‐specific practice effects on self‐administered mobile‐based cognitive tasks in two diverse community‐based cohorts

Kieffer Christianson, Ileana De Anda‐Duran, Cody Karjadi, Lindsay Hathaway, Jose Monteverde, Sherral Devine, Ting Fang Alvin Ang, Phillip H Hwang, Lydia Bazzano, Vijaya B. Kolachalama, Honghuang Lin, Rhoda Au, Preeti SunderaramanAlzheimer 2024 · DOI

TL;DR

This study investigates how practice effects influence performance on mobile-based cognitive tests over a 3-month period in two distinct community cohorts. The findings suggest that more complex tasks show greater improvement due to familiarity, mirroring traditional neuropsychological tests, which could impact the early detection of cognitive decline.

What Problem Does This Paper Address?

The early detection of cognitive decline, particularly in preclinical stages, is a significant challenge in neuroscience and clinical practice. Self-administered mobile-based cognitive assessments offer a promising avenue for frequent, accessible monitoring. However, a key concern with repeated testing is the phenomenon of "practice effects," where performance improves simply due to familiarity with the test, rather than an actual improvement in cognitive function. This can mask or delay the identification of genuine cognitive decline. This paper addresses the question of whether these test-specific practice effects are present in mobile-based cognitive tasks and if they differ across tasks of varying complexity, using data from two community-based cohorts.

Methods

The study utilized three subtests from the Digital Automated Neurobehavioral Assessment (DANA) suite: Code Substitution (CS), Simple Reaction Time (SRT), and Go/No-Go (GNG). These tests were administered via participants' own mobile devices at two time points, 3 months apart (T1 and T2). Cognitive performance was measured using "Cognitive Efficiency" (CE), a composite score reflecting both speed and accuracy. CS assesses visual scanning, attention, learning, and recall; SRT measures pure reaction time; and GNG assesses sustained attention and impulsivity. For each cohort, paired sample t-tests were employed to compare mean CE scores between T1 and T2 for each of the three tests.

Datasets

The study used data from two distinct community-based cohorts:

  • Framingham Heart Study (FHS): A large cohort of 549 cognitively intact participants.
  • Bogalusa Heart Study (BHS): A smaller cohort of 26 cognitively intact participants.

These are existing, well-established longitudinal cohort studies. No new datasets were introduced.

Key Insights and Ideas

  • Practice effects are real and test-specific: The study confirms that repeated exposure to mobile-based cognitive tasks leads to improved performance, a phenomenon known as practice effects.
  • Complexity matters: More complex tasks (CS and GNG) showed more significant practice effects than simpler tasks (SRT). This suggests that the learning and familiarity gained from repeated engagement with more intricate cognitive challenges are more pronounced.
  • Consistency across cohorts: Similar trends in practice effects were observed in both the large FHS cohort and the smaller BHS cohort, lending robustness to the findings.
  • Implications for early detection: The presence of practice effects, especially on more complex tasks, could potentially mask subtle cognitive decline, making it harder to identify individuals at the earliest stages of impairment.

Key Results

  • Code Substitution (CS): Both FHS and BHS participants showed statistically significant improvements in CE scores at T2 compared to T1. The mean difference for FHS was 4.66 (p<0.001), and for BHS was 4.34 (p=0.11).
  • Simple Reaction Time (SRT): No significant improvements were observed for either cohort. The mean difference for FHS was 0.47 (p=0.68), and for BHS was 1.50 (p=0.59).
  • Go/No-Go (GNG): FHS participants showed a statistically significant improvement in CE scores (mean difference 1.96, p=0.007). BHS participants also showed an improvement (mean difference 6.51, p=0.05), which approached statistical significance.
  • Overall: The study found similar trends across both cohorts, with CS and GNG exhibiting more pronounced practice effects than SRT.

Limitations and Future Work

  • Limited time interval: The study only examined practice effects over a 3-month interval. Longer-term effects and the trajectory of these effects are not discussed.
  • Cognitively unimpaired participants: The study focused solely on cognitively unimpaired individuals. How practice effects manifest in individuals with mild cognitive impairment or early dementia is not addressed.
  • Clinical meaningfulness: The authors highlight the need for further research to determine which specific tests are most sensitive to detecting the earliest indicators of cognitive decline and whether the presence or absence of practice effects itself holds clinical significance. Future work should explore these questions and investigate the impact of practice effects on the diagnostic utility of mobile-based cognitive assessments.

Summary updated

Abstract

This study found similar trends in test-specific practice effects on mobile-based cognitive tests in two different cohorts. CS and GNG are inherently more complex than SRT, which may explain why test familiarity improved performance three months after baseline. This is congruent with traditional paper-and-pencil neuropsychological tests, wherein tests with complex stimuli show more practice effects relative to simple tasks. Additional research is needed to determine which tests are more sensitive to detecting earliest indicators of cognitive decline, and whether the presence/absence of practice effects itself are clinically meaningful.
BibTeX
@inproceedings{christianson2024investigating,
  author = {Kieffer Christianson and Ileana De Anda‐Duran and Cody Karjadi and Lindsay Hathaway and Jose Monteverde and Sherral Devine and Ting Fang Alvin Ang and Phillip H Hwang and Lydia Bazzano and Vijaya B. Kolachalama and Honghuang Lin and Rhoda Au and Preeti Sunderaraman},
  title = {{Investigating test‐specific practice effects on self‐administered mobile‐based cognitive tasks in two diverse community‐based cohorts}},
  booktitle = {Alzheimer},
  year = {2024},
  doi = {10.1002/alz.092876},
}
Figure