Brain-Anchored Correction of Practice Effects Across Sites

When people take the same memory test repeatedly, they get better at it simply from familiarity. In aging and early Alzheimer's disease, this learning can cancel out real decline, so a person's scores look flat while their brain is shrinking. Machine learning models that predict cognitive decline from MRI are trained on labels built from these repeated scores. How often patients are retested differs from one hospital to the next. A model can therefore learn a hospital's testing schedule instead of a disease signal, and it will look accurate only when evaluated on data from that same hospital. We propose to use the part of the brain scan that cannot "learn" as a reference. Volumes of structures such as the hippocampus and ventricles are measured by automatic segmentation (software that outlines each brain structure on the scan). They change with disease but not with test familiarity. Comparing score changes against these volume changes lets us estimate the practice effect separately for each site and each number of prior tests, and then remove it before training any predictor.

The payoff is a specific, testable explanation for part of the well-documented gap between internal and external validation in clinical machine learning. The hypothesis is that some of what looks like "scanner shift" or "single-center overfitting" in longitudinal cognitive prediction is actually a site-specific measurement artifact in the targets, not in the images. If that holds, cleaning the targets should shrink the external performance drop by a measurable amount that preprocessing the images alone cannot recover.

Core Claim

Practice effects are large and depend on the cognitive domain. In adolescents, 14% to over 100% of first-to-second-session gains were due to prior test experience rather than growth, and memory composites were the most affected. This comes from Prior test experience confounds longitudinal tracking of adolescent cognitive and motor development, which removed the effect with a "twice-minus-once-tested" contrast: second-time test-takers of age $k$ are compared with same-age first-time test-takers. That design uses age as the clock. In neurodegeneration, age is a poor clock, because two 72-year-olds can be years apart in disease stage. This is exactly why Quantitative detection and staging of presymptomatic cognitive decline in familial Alzheimer's disease flags practice effects as the main confounder that makes early decline look like stability.

Our claim is that segmented regional brain anatomy can replace age as the matching variable. Once it does, practice gains become identifiable in aging cohorts. We would then show that these gains differ systematically across sites, and that correcting for them improves transfer across sites more than image harmonization does. Concretely, for subject $i$ at visit $j$ we model

$$\Delta y_{ij} = f(\Delta A_{ij}, A_{i0}) + g_s(k_{ij}, \tau_{ij}) + \varepsilon_{ij},$$

where $\Delta y_{ij}$ is the change in a cognitive score since baseline, $A_{i0}$ and $\Delta A_{ij}$ are baseline regional volumes and their change measured by segmentation, $k_{ij}$ is the number of prior exposures to the test, $\tau_{ij}$ is the time since the last exposure, and $g_s$ is the practice function for site $s$.

With elapsed time alone, $g$ is nearly inseparable from decline, because both grow with visit number. The anatomy term $f$ absorbs the biological part, so $g_s$ is identified by the residual difference between people at the same anatomical stage who have been tested different numbers of times. ADNI's successive enrollment waves supply the needed first-time testers at the same disease stages.

Research Program

The work has three parts.

The first part builds the anatomical anchor and stress-tests it. We segment hippocampus, amygdala, thalamus and lateral ventricles on ADNI and OASIS-3, using a structural segmentation network of the kind Anatomically Guided Latent Diffusion for Brain MRI Progression Modeling used as a frozen guide. That paper trained on ADNI (31,713 pairs) and evaluated zero-shot on OASIS-3, which is exactly our pair of datasets. It used cognition only as a static diagnosis label, and we treat the cognitive side as the thing that needs correcting.

A known constraint applies here. Acquisition changes shift even global MRI-derived metrics in a model-dependent and diagnosis-dependent way, as Ultra-fast MRI for brain-age prediction in a real-world cognitive disorders clinic showed across six brain-age pipelines. The anchor is only trustworthy if segmentation-derived change is stable when scanners change. We therefore restrict the anchor to within-subject, within-scanner intervals wherever possible. We also flag scanner-change intervals, and we report how much anchor variance sits at those boundaries. This part doubles as a direct answer to the open question of whether anatomical masks themselves stay stable under protocol shift.

The second part estimates $g_s$ per site and per cognitive domain. We use flexible additive models, in the spirit of the generalized additive mixed models in the adolescent study, with inverse-probability weights for dropout. Retesting and attrition are not random: patients who return differ from those who do not, a point made in Di Wang et al. and acknowledged by the 13-year cohort of Threshold Effects, Cognitive Decline, and Longitudinal Changes in White Matter Hyperintensity Volume.

The adolescent data give an internal validity check. The estimated practice function should be large for memory-heavy scores (ADAS13 word recall) and small for measures closer to processing speed. If the anchor-based estimate does not reproduce this domain ordering, it is capturing something other than learning.

The third part asks what the correction buys for prediction. We rebuild longitudinal targets, namely corrected slopes of ADAS13 and MMSE and a corrected conversion label, and train the same ROI-based predictors on raw versus corrected targets. We evaluate each predictor three ways: internally within a single ADNI site, under leave-one-site-out testing across ADNI, and zero-shot on OASIS-3.

The single-site arm is deliberate. It recreates the situation of A multimodal MRI-based machine learning framework for classifying cognitive impairment in cerebral small vessel disease. That study reported 0.878 AUC on an internal validation cohort drawn from the same hospital as its training data, and its authors acknowledged it was unverified externally. The 3-year follow-up labels in that study used repeated MoCA/MMSE testing with no practice adjustment. We predict that within one site the testing schedule is homogeneous, so practice is silently absorbed into the labels and internal scores stay high. Across sites, differing schedules turn the same absorption into error.

Baselines

The naive reference is a predictor trained on uncorrected longitudinal scores and uncorrected clinical conversion labels, evaluated under internal random splits. This is the default practice the reviews by Spohn et al. and Stewart et al. identify as over-optimistic. The same predictor evaluated leave-site-out gives the size of the external gap that we aim to explain.

The strongest existing predictive method on this data is Brain Region-Centered MultiModal Hypergraph Fusion for MCI Conversion Prediction. It builds brain-region features from an anatomical atlas, fuses them with MRI, PET and clinical scores in a hypergraph, and reports 89.7% AUC, 80.71% accuracy and 82.71% balanced accuracy on 479 ADNI participants, using uncorrected MMSE and ADAS13 as inputs within ADNI. We will re-implement it with its published inputs. It is then trained on raw and on corrected targets, with the same splits and the same hyperparameter search budget for each condition, and evaluated leave-site-out and on OASIS-3. Its cognitive-score inputs will also be replaced by their corrected versions.

We do not expect to beat 89.7% AUC internally. The informative result is the gap between internal and external performance. We would count as a win a reduction of that gap by at least one third, with non-overlapping bootstrap intervals, and no loss of external AUC. For progression, no reported result was found on Lacuna for OASIS-3 numerical performance of the anatomically guided diffusion model beyond the statement that volumetric error "remained low." Its 15–20% volumetric error reduction over BrLP is a within-ADNI figure and serves only to justify reusing its segmentation guidance.

Two ablations isolate the key ingredient. In the first, the practice function is estimated with age matching instead of anatomy matching, which is the direct transfer of the adolescent method. In the second, a single pooled practice function replaces the site-specific $g_s$. If the anatomy-matched version does not beat the age-matched one on the checks below, the anatomical anchor adds nothing. If pooled and site-specific corrections give the same external gain, the site-dependence part of the claim is false.

Why not just…?

Why not just add visit number as a covariate in a mixed model? This is the most serious alternative, and it may work in cohorts where retest intervals vary a lot, because interval variation partly separates learning from elapsed time. It fails where visit number and time move together, which is the norm in scheduled consortium follow-up. There, a visit-number coefficient soaks up real decline and under-corrects. We run it as a baseline under the same weighting. If it matches the anatomy-anchored estimate both in domain ordering and in the external gap reduction, we will report that the simpler correction suffices and that the contribution reduces to showing that practice correction closes the external gap.

Why not just subtract published Reliable Change Index norms? The Reliable Change Index is a published formula for the change expected from retesting alone. The norms in Normative data for baseline and longitudinal neuropsychological assessments in Singapore and the short-interval thresholds of Test-Retest Reliability, Convergent Validity and Practice Effects of the RBANS in a Memory Clinic Setting come from particular populations and fixed intervals. They supply one expected gain, whereas our hypothesis is that the gain varies by site, interval and disease stage. People with failing memory consolidation show smaller practice gains, so a fixed subtraction over-corrects in exactly the patients who matter. Applying a fixed practice subtraction is therefore another baseline, and the site-specific versus pooled ablation tests the difference directly.

Why not just predict imaging change and skip cognition entirely? Imaging targets sidestep practice, but clinical endpoints, trial outcomes and conversion labels are cognitive. Imaging targets also carry their own acquisition shift, as the brain-age results show. We do report imaging-only prediction as a reference point.

Why not just harmonize the images (for example with ComBat)? ComBat is a statistical method that removes site-specific offsets from image-derived features. We apply it in every arm. Our prediction is that it cannot fix a problem located in the labels. If image harmonization alone closes the same external gap that label correction closes, our central claim is refuted.

Decisive Checks

Five results decide the claim. First, the anchor-estimated practice function must show large gains for memory scores and small ones for speed and motor measures, matching the adolescent evidence. Second, estimated gains must differ across ADNI sites and OASIS-3 by more than bootstrap noise, and the differences must track differences in retest schedules. Third, the internal-to-external AUC gap for both the naive predictor and the hypergraph baseline must shrink under corrected targets beyond what harmonization achieves, and beyond what the age-matched and pooled ablations achieve. Fourth, in single-site training, the fraction of the external drop explained by differences in exposure schedule between the training site and each test site should be positive and significant in a site-level regression.

The fifth check is secondary and can be run wherever white matter hyperintensity (WMH) segmentations are available. WMH are small bright lesions on MRI that mark vessel disease. We test whether the "paradoxically positive" link between low WMH burden and cognition reported in the 13-year cohort shrinks after practice correction. This would distinguish test familiarity from the neural compensation its authors proposed.

A clean negative on the first three checks would also be useful. It would show that practice effects, though real, are not a meaningful driver of cross-site failure in these benchmarks.