Models that detect cognitive impairment from brain scans and test scores usually lose accuracy when moved from the hospital that trained them to a new one. The standard explanation is technical: different scanners, different imaging protocols, different patient mixes. This note proposes a third explanation that has been largely overlooked. Each memory clinic is itself an observer, and clinics draw the line between "normal aging," "mild cognitive impairment," and "dementia" in different places. If that is true, part of the cross-site performance drop is not the model failing to see the brain correctly. It is the model being scored against a different diagnostic convention. This boundary disagreement should be largest exactly where population aging puts the most pressure on clinics: very old, less educated, functionally dependent patients, whose brain shrinkage and test scores fall in the overlap between normal aging and disease.
We propose to measure this effect directly. The plan treats each center as a rater, borrowing tools built for disagreement among individual pathologists. It splits the external-validation loss of cognitive-impairment models into three parts: acquisition differences, population differences, and labeling differences. It then tests whether a small, interpretable per-center correction, one that is allowed to depend on age and education, recovers calibrated performance at unseen centers. The answer matters for deployment in rapidly aging health systems. If the gap is mostly labeling, the right response is to report and adapt to a clinic's diagnostic convention, not to keep harmonizing images.
Core Claim
Our claim is that in multi-center dementia cohorts, some of the measured generalization gap reflects center-specific diagnostic conventions, that is, a change in how labels are assigned given the patient's features. We further claim that this convention is not a single constant offset. It varies with patient age and education, so the oldest and least educated patients see the largest cross-center disagreement. Both parts are falsifiable. If the labeling share of the gap is negligible, or if a single per-center intercept removes it, we will report that, and the simpler fix wins.
Three findings in the file motivate this. First, The Latent Doctor Model infers an individual pathologist's diagnostic style from a small set of that pathologist's labeled cases, using a DeepSet encoder (a network that summarizes an unordered set of examples into one vector). It shows that modeling style, rather than averaging it away, matches panel disagreement, reaching KL divergence 0.461 on Gleason2019 and Spearman ρ up to 0.728 between model entropy and panel entropy on ISUP. That work stays within single-institution pathology datasets. Second, the virtual expert panel work trains one model per pathologist and finds that consensus training badly miscalibrates predictions. Third, multi-center cognitive models such as Kang et al. and Romano et al. pool diagnoses from several centers as if they were a single ground truth. The step we add is to move the "rater" from the individual doctor to the institution. We can do this without collecting any new multi-rater labels, because every center in a consortium cohort already acts as a separate diagnostic observer.
The aging context does two jobs in this design. It defines the regime where we expect boundary disagreement to concentrate, and it supplies the stratification axes. Guo et al. report that educational background distorts MMSE screening in older adults, and that functional decline among empty nesters is driven partly by non-cognitive factors such as anxiety and polypharmacy. Both findings suggest that cognitive labels in older, more isolated populations depend heavily on how a clinician reads the case. We do not claim to model living arrangements causally. Guo et al. is a single-region cross-sectional survey without imaging, and the file does not confirm which social covariates the imaging cohorts record.
Research Program
The model has a shared disease score and a small, interpretable center-specific head:
$$p(y \ge k \mid x, a, c) = \sigma\big(f(x) - \theta_k + b_c + g_c(a)\big)$$
Here $x$ is the patient's imaging and cognitive-test features, and $f(x)$ is a disease score shared across all centers. The terms $\theta_k$ are ordinal cut-points for the categories normal, MCI, and dementia. The symbol $a$ stands for age, education, and functional-dependence covariates. For center $c$, $b_c$ is a scalar offset, and $g_c(a)$ is a smooth additive function of $a$ that captures age- and education-dependent shifts in where that center draws its boundary.
Following the reluctant additive idea of RAITR, which starts from a sparse linear model and adds penalized nonlinear terms only when they explain leftover variance, we penalize $g_c$ more heavily than $b_c$. That way the age-dependent convention enters the model only if the data demand it. For a new center, $b_c$ and $g_c$ are inferred from a small support set of that center's labeled cases. We do this either by direct penalized fitting or with an LDM-style DeepSet encoder, and we compare the two. Because the head is additive, each center's convention can be read off as a plot: "this clinic labels an 85-year-old with 8 years of schooling as MCI at a disease score 0.4 lower than the pooled average."
The central analysis is a decomposition of the external-validation loss. For a source-trained model evaluated at a target center, we can apply three corrections one at a time. We correct population mix by reweighting source patients to match the target's age, education, and functional-dependence distribution. We correct labeling convention by substituting the target's fitted $b_c, g_c$. We correct acquisition by using within-center paired or same-scanner subsets where they exist. Because the order of corrections matters, we average each factor's contribution over all six orderings, which is a three-player Shapley value. This gives an attribution of the gap that does not depend on an arbitrary sequence.
Labeling style and true disease differences are confounded unless we have a reference that does not depend on any center's convention. We use two such anchors. The first is neuropathology from NACC, which the file describes as including neuropathological data. The second is longitudinal outcome, meaning later conversion or stability, as used in Romano et al.'s survival framing. A center whose labels are systematically "early" relative to both anchors is showing a convention, not a sicker population.
Finally, we connect this to interpretability. Using the Disease-Focus score from the DF-score paper maps gradient saliency onto FastSurfer brain segmentations and scores whether the top regions are core Alzheimer's pathology. We ask two questions with it. Does a shared $f(x)$, trained with center heads absorbing the convention, attend more to hippocampus and entorhinal cortex than a pooled model? And do models trained on individual centers' labels show different anatomical focus? A pooled model forced to fit conflicting boundaries may learn to use age-related atrophy as a shortcut.
Baselines
The naive reference is a pooled model trained on all source centers' diagnoses as one ground truth, evaluated as-is at held-out centers. This is the setup of Kang et al. and Romano et al. For reference, Romano et al. report an integrated Brier score of 0.192 on ADNI and 0.108 on external NACC for their survival CNN. The DF-score paper reports F1 0.92 and balanced accuracy 0.81 on external AIBL for MedicalNet with augmentation, along with a Disease-Focus score of 1.90. We will re-run a pooled 3D CNN and a pooled gradient-boosted tabular model on our own splits rather than compare against these figures directly, because the tasks and splits differ.
The strongest simple competitor is per-center recalibration. This means fitting only a per-center intercept, or Platt scaling, on the same support set we give our model. It is the special case of our model with $g_c = 0$, so it isolates whether the age-dependent part matters. Our model beats it only if it improves negative log-likelihood and ECE, especially in the 85+ and low-education strata, at matched support-set size.
The strongest existing multi-rater method is the virtual expert panel adapted so that each source center is a "rater," with the per-center models ensembled. That work reports ECE falling from 0.183 for consensus training to 0.050 for the full panel (Grove et al.). We also run an LDM with a center-level DeepSet encoder. For both, there is no reported result found on Lacuna for dementia diagnosis or for centers as raters. All methods share the same backbone, features, training data, support sets, and hyperparameter search budget.
The key ablations are center labels randomly shuffled, which should erase any gain if the latent truly captures center convention, and support-set size varied from about 10 to 200 cases.
Why not just…?
Why not just add site as a covariate or recalibrate per center? This is the real threat, and it is a baseline above. If a single offset per center closes the gap, then centers differ only by a constant threshold, and our richer head is unnecessary. The experiment settles this directly. We compare intercept-only and age-dependent heads on held-out centers, stratified by age band. We expect the advantage of the richer head, if any, to appear only in the oldest and least educated strata. If it does not appear there, we will report recalibration as the answer.
Why not just harmonize images or augment harder? Harmonization and augmentation act on how features are distributed. They cannot change how a clinic maps a given brain to a label. The file also shows they can mislead. In the DF-score paper, augmentation improved external accuracy without improving disease focus. In the ultra-fast MRI study, changing protocol within a single clinic produced model-specific brain-age offsets and acquisition-by-diagnosis interactions. Our decomposition reports how much of the gap the acquisition factor accounts for. If it dominates, harmonization is the right tool, and our study will say so.
Why not just use tabular cognitive scores with a simple tree model? Haddada et al. obtained 95% accuracy with decision trees on clinical features in a single cohort, versus 81% from MRI volumes. However, labels in such cohorts are partly defined by those same test scores, which inflates accuracy and makes the setup circular. It also cannot separate a center's convention from the patient's status. We therefore run the full analysis on tabular features too, including a replication on Kang et al.'s three-center neuropsychological battery. In that replication, the image-based Disease-Focus analysis drops out but the labeling decomposition still applies.
Why not just audit subgroups with AFISP? AFISP adversarially searches for patient subgroups where a model underperforms, and found AUROC falling from 0.986 overall to 0.81 in one subgroup. It finds where a model fails, but it treats the labels as fixed. We will run AFISP on the pooled model and check whether the subgroups it flags line up with the strata where our fitted center conventions disagree most. If they do, that means at least some "underperforming subgroups" are really disagreements about labels.
Proposed Experiments
The main testbed is leave-one-center-out evaluation across NACC's Alzheimer's Disease Centers, with ADNI as an additional source. ADNI's diagnoses follow protocol-defined cut-offs and so behave like a near-algorithmic "rater." AIBL and OASIS-3 serve as further external centers, and Kang et al.'s three-center battery serves as a tabular replication. Tasks are three-way normal/MCI/dementia classification and MCI-to-dementia conversion. Metrics are negative log-likelihood, ECE, AUROC, and integrated Brier score for conversion, all reported overall and by age band (60–74, 75–84, 85+) and education tertile. We also report the three-way Shapley decomposition of the gap, agreement of fitted conventions with the neuropathology and conversion anchors, and the Disease-Focus score for shared versus pooled backbones.
Three checks decide the claim. First, the labeling share of the external gap must be clearly nonzero, with bootstrap intervals over centers that exclude zero. Second, each center's fitted convention should predict where its labels fall relative to autopsy and later conversion, and this effect should vanish when center labels are shuffled. Third, the age-dependent head should beat per-center recalibration at matched support size, specifically in the oldest strata. If the first two checks hold but the third fails, the contribution narrows to "report and recalibrate per-center conventions," which is still useful for deployment. If the first fails, the multi-center gap in dementia models is mostly technical or demographic, and the harmonization literature is aimed at the right target.
References
- The Latent Doctor Model for Modeling Inter-Observer Variability
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Prediction of cognitive impairment via deep learning trained with multi-center neuropsychological test data
- Deep learning for risk-based stratification of cognitively impaired individuals
- Prevalence and factors influencing disability and cognitive impairment among empty nesters and non-empty nesters in Guangdong, China: a cross-sectional study
- A reluctant additive model framework for interpretable nonlinear individualized treatment rules
- A Quantitative Approach for Evaluating Disease Focus and Interpretability of Deep Learning Models for Alzheimer's Disease Classification
- Ultra-fast MRI for brain-age prediction in a real-world cognitive disorders clinic
- Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease
- A data-driven framework for identifying patient subgroups on which an AI/machine learning model may underperform