Separating Real Cognitive Benefit from Practice and Self-Selection in Gamified Brain Training

Gamified cognitive training apps, which turn cognitive exercises into games with scores, levels and difficulty that adapts to the player, report striking adherence and score gains in older adults at risk of dementia. Those gains are hard to interpret. A player's in-game score rises partly because they get better at the game, which is a practice effect, and only partly because their cognition improves. The game makes itself harder when they do well, so today's performance shapes tomorrow's exposure. People who feel themselves slipping may play more, or may quit out of frustration, so the amount of play is chosen in part by the very decline it is supposed to treat. Studies also stop after a few months, before novelty wears off. Together these problems mean that "people who played more improved more" cannot currently be read as evidence that the game works.

This note proposes a study design and analysis that make the question answerable. It rests on two ingredients. The first is an untrained anchor: a cheap, at-home measurement of cognitive state, brief recorded speech scored with acoustic and lexical features, that the game never practices. Because of that, its trajectory cannot be inflated by game familiarity. The second is a small randomized element embedded in the app: randomized play reminders and randomized difficulty schedules. These give an experimental yardstick against which the observational correction for self-selected engagement can be scored. A planned washout, in which play stops while anchor measurement continues, tests whether gains persist or were a honeymoon effect.

Core Claim

The main claim is that, in an adaptive gamified cognitive therapy, three effects can be told apart: the causal effect of engagement on cognitive trajectory, task familiarity, and selection or feedback bias. Two conditions are required. Cognitive state must be measured through a channel that loads on cognition but not on cumulative gameplay, and some variation in engagement must be randomized so that the de-confounding method can be checked rather than trusted. Neither condition is met in existing work. Without them, every reported dose-response relationship in this literature is uninterpretable.

The measurement assumption can be written as a small latent-variable model. Let $\theta_i(t)$ be person $i$'s latent cognitive state at time $t$, $n_{it}$ their cumulative number of game sessions, $Y_{it}$ their in-game performance and $S_{it}$ their speech-anchor score:

$$ Y_{it} = \theta_i(t) + p(n_{it}) + \epsilon_{it}, \qquad S_{it} = \lambda\,\theta_i(t) + u_{it}. $$

Here $p(\cdot)$ is a practice curve, $\lambda$ is the anchor's loading on cognition, and $\epsilon_{it}$ and $u_{it}$ are measurement noise. From in-game scores alone, $\theta_i(t)$ and $p(n_{it})$ are both increasing in time for engaged players, so they cannot be separated. The anchor breaks this tie because it contains no $p$ term. That exclusion is an assumption, and the proposal tests it directly in untreated controls.

Relation to Prior Work

The closest work is ADL+, a gamified multidomain toolkit for community-dwelling older adults. It reported 95% median adherence and a 0.17 between-group gain in a composite of neuropsychological test scores at 6 months. By 9 months only processing speed remained superior. The authors acknowledge three limits: baseline group imbalance in a quasi-experimental design, a 3-month post-intervention window too short to judge durability, and no independent biomarker to rule out task familiarity. This proposal is built around exactly those three admissions. LETGAMES generates personalized game content with large language model agents, but it was evaluated only with simulated patients. It motivates the concern that adaptive systems tie exposure to performance, and it supplies nothing longitudinal.

On the measurement side, several papers supply the anchor. Acoustic features from recorded neuropsychological exams reached AUC 0.92 (0.90 with acoustics alone, against 0.79 for demographics) in a small Framingham sub-cohort (Spoken language biomarkers). At scale, acoustic composite scores raised prevalent mild cognitive impairment (MCI) AUC from 0.712 to 0.755, and seven composites were associated with incident MCI (Framingham acoustic study). Other results constrain the design. On the ADReSS benchmark, the best acoustic-only test accuracy was just 0.625 (ADReSS Challenge). Separately, simple context-free word embeddings with a naive Bayes classifier reached 90% and beat Bio-Clinical BERT at 84% (Who needs context?). The anchor will therefore combine acoustic features with lightweight lexical features, and it must be validated before use rather than assumed. The very high 97.42% accuracy of an attention-based hybrid network on the Pitt corpus (attention hybrid) comes from a single picture-description task. That result does not establish sensitivity to month-to-month change, which is what this study needs. DANA shows that repeated smartphone reaction-time tasks track MCI but suffer practice effects. It illustrates why a game-derived score cannot serve as its own outcome.

On the causal side, the analysis method comes from NLME standardization. That paper shows that when early outcomes drive later adherence, both intent-to-treat and per-protocol analyses are biased, while conditioning on individual random effects recovers the causal effect. It has only been tested on 25,000 simulated patients with a correctly specified parametric model. Here it is applied to real digital telemetry, and the randomized nudges check whether its assumptions hold. Doubly robust estimation with targeted minimum loss-based estimation, as in the olanzapine–aripiprazole study, and the IPTW-based target trial emulation show that careful baseline adjustment can overturn naive conclusions. Both adjust only for baseline confounders, however, and cannot handle performance-driven feedback. The SHARP psychosis cohort is a cautionary example: confounding by indication was documented but could not be removed with ANOVA and stratification.

Research Program

The study is a prospective cohort of roughly 400 community-dwelling older adults at elevated risk of decline. All participants receive an adaptive gamified cognitive training app modeled on ADL+-style exercises. They play for 9 months, followed by a 6-month washout in which the app is locked and measurement continues. Fifteen months is still short for clinical conversion. The design therefore does not claim a dementia endpoint. Its target is the effect on the anchor trajectory and on its persistence after play stops.

Every two weeks, participants record about three minutes of speech at home through the same phone. They do a picture description and a short free-recall narrative, using prompts rotated so that the speech task is not itself practiced. A conventional neuropsychological battery is given at months 0, 9 and 15. Plasma p-Tau217 is drawn at baseline in the subset where it is feasible, which allows stratification by underlying pathology. That marker predicted 4-year MCI-to-Alzheimer's conversion with AUROC 0.787 in ADNI (survival analysis).

Two layers of randomization are embedded. First, each week every participant is randomized to receive or not receive push reminders. This shifts engagement for reasons unrelated to cognition and works as an instrument for dose. Second, at enrollment participants are randomized between two difficulty-adaptation rules: one that follows performance and one on a fixed schedule. This lets us measure how much performance-driven feedback alone distorts dose-response estimates.

The speech anchor model is fit beforehand on external data: ADReSS for classification, and the published Framingham feature definitions for acoustic composites. Its weights are then frozen, so it is never tuned on study outcomes. This avoids the circularity the file notes, where composite weights are derived from the same tests that define diagnosis. The primary analysis fits the latent model above with person-level random slopes for $\theta_i(t)$. Engagement dose acts on the slope. Time-varying engagement is handled through standardization over random effects, which applies the NLME result to observational variation in dose.

Baselines

The naive reference point is the analysis the field currently reports: regress change in in-game score, and separately change in the battery score, on total sessions played, adjusting for baseline age, sex and education. We expect it to show a strong positive dose-response. Much of that will be practice and selection.

The second baseline is intent-to-treat and per-protocol analysis of the randomized reminder arm. This is the comparison that the NLME paper showed to be biased under feedback in simulation. Here it is checked against an instrumental-variable estimate.

The strongest existing observational competitor is doubly robust TMLE with a Super Learner, adjusting for baseline covariates including baseline anchor score and, where available, p-Tau217. It follows the olanzapine–aripiprazole study. That study is the strongest baseline because it overturned a long-standing naive conclusion in 21,293 patients. It reports 24-month RMST differences of 0.16 to 0.25 diabetes-free months and E-values of 1.5 to 3.0. Those figures concern a different outcome and are not a performance benchmark for this question. No reported result was found on Lacuna for de-confounded estimates of engagement effects in gamified cognitive therapy.

All methods use the same data, the same covariate set and the same pre-registered outcome. Every flexible learner gets identical tuning budgets, using the same cross-validation folds and the same Super Learner library. The proposal "beats" a baseline if two things hold. Its observational estimate of the per-session effect on anchor slope must fall within the 95% confidence interval of the instrumental-variable estimate from randomized reminders. The baseline's estimate must fall outside that interval.

Two ablations isolate the key ingredients. The first refits the full model with in-game scores and no anchor; this tests whether the anchor is what separates practice from benefit. The second fits the anchor model while ignoring the randomized difficulty arm; this tests how much feedback-induced bias the adaptive rule introduces. A third ablation replaces the speech anchor with the sparse three-visit battery.

Why not just…?

Why not just run a long randomized trial of the app against a control? For the narrow question "does assigning the app help?", a long trial is the gold standard. We do not claim otherwise. It does not answer how much engagement helps, which is what adaptive dosing and clinical advice need. Its intent-to-treat effect is also diluted by the same attrition that plagues digital trials. The embedded reminder randomization is in fact a small randomized trial of dose, and it is cheap. If a funder can support a multi-year trial, this design nests inside it unchanged.

Why not just model practice in the in-game scores, for example by adding a session-count term? Practice and real change both increase with time among engaged players, so a session-count term cannot be separated from a cognitive slope without an outside measurement. The DANA work (DANA) notes this masking explicitly. The no-anchor ablation tests it directly. If that ablation recovers the instrumental-variable estimate as well as the full model, the anchor is unnecessary, and we will report that.

Why not just adjust for baseline severity with propensity weighting or TMLE? Baseline adjustment cannot address confounding that develops during play. A participant who struggles in month two, gets easier levels and then disengages creates bias that no baseline covariate captures. The simulations in the NLME paper show this, and the SHARP cohort shows that residual confounding by indication persists in practice. The comparison against the instrumental-variable estimate settles whether TMLE is good enough here.

Why not just use the standard neuropsychological battery as the anchor? This is a real threat. Batteries are better validated than speech features, which performed weakly in ADReSS acoustic-only tests at 0.625 accuracy. But batteries need clinic visits, have their own practice effects and can only be given a few times, which makes slopes noisy. The battery-anchor ablation decides the question. If it matches the speech anchor in agreement with the instrumental-variable estimate, the speech anchor's only contribution is accessibility and measurement density, and the paper will say so.

Decisive Checks

The anchor exclusion assumption is checked first. It holds if, among participants randomized to fewer reminders and matched on baseline, anchor slope is unrelated to cumulative sessions before any plausible treatment effect could occur. It also requires that test–retest change in the anchor over the first month shows no learning trend. The anchor must reach adequate held-out discrimination on ADReSS before it is used; the target is at least the 0.75 linguistic baseline reported by the challenge.

The central check is the within-study comparison against the randomized reminder estimate. It holds if the anchor-plus-NLME estimate of the effect of engagement on anchor slope agrees with the reminder-based estimate, and the naive and baseline-TMLE estimates do not.

The washout tests durability. If anchor gains decay to control levels within six months of locking the app, the result is evidence of a honeymoon effect rather than disease modification. That would be an important negative result for the field. If gains persist, and especially if they persist among p-Tau217-positive participants, the case for gamified training as more than practice becomes much stronger than anything in the current literature.

References

  1. ADL+: A Digital Toolkit for Multidomain Cognitive, Physical, and Nutritional Interventions to Prevent Cognitive Decline in Community-Dwelling Older Adults
  2. LETGAMES: An LLM-Powered Gamified Approach to Cognitive Training for Patients with Cognitive Impairment
  3. Spoken Language Biomarkers for Detecting Cognitive Impairment
  4. Association Between Acoustic Features and Neuropsychological Test Performance in the Framingham Heart Study: Observational Study
  5. Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge
  6. Who needs context? Classical techniques for Alzheimer's disease detection
  7. An Attention-Based Hybrid Network for Automatic Detection of Alzheimer's Disease from Narrative Speech
  8. Effectiveness of Self‐Administered Mobile Assessment in Detecting Mild Cognitive Impairment
  9. Correcting for confounding in longitudinal experiments: positioning non-linear mixed effects modeling as implementation of standardization using latent conditional exchangeability
  10. Revisiting diabetes risk of olanzapine versus aripiprazole in serious mental illness care
  11. Target trial emulation: Do antimicrobials or gastrointestinal nutraceuticals prescribed at first presentation for acute diarrhoea cause a better clinical outcome in dogs under primary veterinary care in the UK?
  12. Cognitive functions following initiation of antipsychotic medication in adolescents and adults at clinical high risk for psychosis: a naturalistic sub group analysis using the MATRICS consensus cognitive battery
  13. Predicting Progression from Mild Cognitive Impairment to Alzheimer’s Disease using Survival Analysis