Difference-First Temporal Fusion for Cross-Cohort Disease Progression

A model that forecasts dementia from brain scans is usually trained at one consortium of hospitals and then fails elsewhere. It fails for two reasons that look the same on a scoreboard. One is physical: a different scanner makes every image a little brighter, blurrier or differently contrasted. The other is biological: the new population progresses differently. An example is the heavier vascular disease burden in a Singapore memory clinic. Current fixes treat both as nuisance "style" and pull the new site's features toward the training site. That removes the scanner effect, but it also erases real differences in how disease unfolds in the new population.

This note proposes a simple ordering principle for models that read a patient's series of scans: compute within-patient change first, and look at anatomy across patients last. A patient is usually scanned on the same machine across nearby visits, so most scanner distortion is shared by both scans in a pair and largely cancels when the model's first stage compares them. Later stages then fuse these short-range changes into longer trajectories. At no point does the model force populations to look alike. We claim this "difference-first" hierarchy closes much of the scanner part of the cross-cohort gap without any alignment step, and leaves population-specific progression intact. We also propose an evaluation that splits the transfer gap into its scanner and population parts, so the claim can be checked directly.

Core Claim

Hierarchical temporal fusion means building up the time window gradually across network stages instead of merging all time points at the input or only at the output. It was developed for dense video, as in Large-Scale Video Classification with Convolutional Neural Networks, which merged neighboring frames layer by layer, and A Spatial-Temporal Progressive Fusion Network for Breast Lesion Segmentation in Ultrasound Videos, which used past ultrasound frames as memory for the current one. Our claim is that in sparse longitudinal imaging, the hierarchy's value is not mainly a larger receptive field. Its value is invariance. If the first fusion stage sees only pairs of a patient's own scans, it is insensitive to any distortion that stays constant within that patient. A rough model is an observed scan $\tilde{x}_k = g_s(x_k)$, where $x_k$ is the true anatomy at visit $k$ and $g_s$ is a site-specific scanner transform. If $g_s$ is close to an additive or smooth monotone intensity change, then a pair encoder that compares $\tilde{x}_k$ with $\tilde{x}_{k+1}$ sees roughly $g_s$-free change. A cross-sectional encoder of $\tilde{x}_k$ sees $g_s$ directly.

The claim makes two predictions that can be wrong. First, the advantage should disappear exactly where its assumption fails: visits where the patient's scanner changed, which happens in ADNI across protocol phases and field strengths. Second, a difference-first model should transfer better than feature-alignment methods to a biologically different cohort, because alignment pushes that cohort's trajectories toward the majority cohort's.

How This Relates to Prior Work

Three papers motivate the gap. Cross-dataset Evaluation of Dementia Longitudinal Progression Prediction Models trained progression models on ADNI and tested them on AIBL, MACC and OASIS. It found real transfer gaps and found that L2C-FNN was the strongest model. L2C-FNN is a feedforward network that flattens irregular visit histories of regional volumes into one vector. The study never separates scanner effects from population effects and never tries adaptation. UG-FedDA treats differences between ADNI, AIBL and OASIS as style to be aligned, down-weighting uncertain samples. It works only on single-visit classification. A cross population study of retinal aging biomarkers uses temporal order as a pretraining signal across UK and Chinese cohorts. It still predicts from one image and still adds explicit cross-domain alignment.

Two papers supply method components. Temporal Flow Matching showed that modeling change rather than the whole volume is what lets a model beat the "copy the last scan" baseline. It was evaluated only within single cohorts, and it fills missing visits by repeating the last scan. We reuse its difference idea and give it a different job: invariance to scanner, not just escaping copy-the-last-scan. Longitudinal Pooling & Consistency Regularization introduced a loss forcing disease probability to be non-decreasing. We adopt that constraint, because scanner noise can otherwise produce fake reversals of atrophy.

Two papers constrain the design. Adapt the Data, Not the Model found that cross-hospital shift in ICU time series was driven by recording cadence, not by gaps between feature distributions. Its adapted data actually moved further from the source in Wasserstein distance while task performance rose. That warns us not to judge success by distribution distance. Fair CCA reduced ADNI sex disparity by linear projection, but at a cost in ROC-AUC. That illustrates the accuracy loss from removing attributes that carry biology. We therefore measure subgroup gaps rather than projecting attributes out.

Research Program

The model has three stages and one late branch.

The first stage is a pair encoder. It takes two adjacent visits of the same patient, rigidly registered, and outputs a change embedding divided by the time gap. This is a rate of change, so irregular intervals are handled by construction rather than by forward-filling missing visits. Each pair carries a flag saying whether the scanner model or field strength changed between the two visits.

The second stage fuses neighboring rate embeddings within short windows of two or three intervals. This captures acceleration. The third stage applies attention over all windows, with continuous time encoding, to produce a trajectory summary.

The late branch encodes the baseline scan itself. Static anatomy matters for prognosis, but it is the channel through which scanner style enters. This branch joins only at the prediction head, so it can be ablated or downweighted.

The heads predict future clinical diagnosis, MMSE (a cognitive test score) and ventricle volume, following the setup of the cross-dataset dementia study. A monotonicity loss applies to diagnosis probability and ventricle volume. We build two versions of the same architecture to keep the principle separate from the input type. One runs on 3D T1 MRI. The other runs on the same regional volumes that L2C-FNN uses, with pairwise differences fed to a small first stage.

The analysis component is a gap decomposition. Within ADNI we compare held-out patients with stable scanners against those whose scanner changed. This isolates scanner shift with the population held fixed. Across cohorts, we compare ADNI-to-AIBL, ADNI-to-OASIS and ADNI-to-MACC on a subset matched in scanner field strength. This isolates population shift as far as the data allow. Population bias is reported as per-cohort and per-sex error and calibration gaps, alongside overall accuracy. MACC access is restricted. If it cannot be obtained, AIBL and OASIS remain, and the population claim is weakened accordingly.

Baselines

The naive reference is persistence. It carries the last observed diagnosis, MMSE and ventricle volume forward, and it is the image-level analogue of the last-context-image baseline. Temporal Flow Matching reports that this baseline is hard to beat: on Lumiere, last-context NRMSE was 0.085 versus 0.069 for flow matching and 0.711 for SimVP (Temporal Flow Matching). Beside it we run a single-visit 3D CNN. Longitudinal Pooling & Consistency Regularization reports 86.1% balanced accuracy on ADNI for a single-timepoint CNN.

The strongest existing longitudinal image model is the CNN+GRU with longitudinal pooling and consistency loss from that same paper, which reports 90.4% ADNI balanced accuracy. It is the natural competitor because it already uses all visits and the monotonic constraint. It differs from ours only in flat late fusion of per-visit features. We re-run it on our splits, removing future-visit pooling to prevent information leakage.

The strongest cross-cohort system is L2C-FNN from Cross-dataset Evaluation of Dementia Longitudinal Progression Prediction Models. We found no reported numerical result on Lacuna for L2C-FNN on AIBL, MACC or OASIS, only the qualitative statement that it was most accurate across external datasets. We will therefore reproduce it with its released features and training on ADNI alone.

For alignment, we apply UG-FedDA-style uncertainty-weighted feature alignment to the CNN+GRU, without the federated setting. Its reported static accuracies, for example 90.54% on ADNI and 77.78% on OASIS for NC versus AD, are for a different task. They serve only as context, not as targets.

All models get the same patients, splits, registration, input resolution, and the same hyperparameter search budget, counted in trials and GPU hours. Model selection uses ADNI validation data only, so no external-cohort information leaks in.

Beating the field means a lower external-cohort error than both reproduced L2C-FNN and the CNN+GRU, beyond the spread across seeds and patient bootstraps. It also means no larger cohort or sex gaps, and better results than persistence on MMSE and ventricle volume at every forecast horizon.

The key ablations are these. A flat difference model computes all pairwise differences and pools them in one step, with no hierarchy. A cross-sectional-first hierarchy fuses per-visit embeddings rather than pair differences. A version drops the late anatomy branch. A version drops the scanner-change flag.

Why not just…?

Why not just use L2C-FNN on regional volumes, which already transfers well? This is the real threat. FreeSurfer-style volumes are partly harmonized already, and their differences across visits may already provide the cancellation we claim. That is why we build a regional-volume version of our model. If difference-first fusion on regional volumes matches the image model, the honest conclusion is that the principle holds and end-to-end imaging is unnecessary. That would still be a useful result. If neither beats L2C-FNN, the claim fails.

Why not just harmonize or align features across sites? The input-adaptation results in Adapt the Data, Not the Model show that reducing distribution distance is not what improves clinical prediction. The cross-dataset dementia study documents biological differences, such as the vascular burden in MACC, that alignment would suppress. We test this directly: if the alignment baseline transfers as well as ours to MACC, with no extra error in vascular or high-risk subgroups, our population argument is wrong.

Why not just train on several cohorts at once? Pooled training is the right comparison when target data exist. We run it as an upper reference. Our setting is training on one source cohort and deploying to new ones, which is how certified models are usually shipped. We also report whether difference-first fusion narrows the gap between source-only and pooled training.

Why not just a single pair-difference model without hierarchy? The flat difference ablation answers this. If it matches the full model, the contribution is the differencing, not the hierarchy, and we will say so.

Proposed Experiments and Decisive Checks

The main experiment trains each model on ADNI and forecasts diagnosis (balanced accuracy and AUC), MMSE and ventricle volume (mean absolute error) on AIBL, OASIS and MACC, at horizons of up to six years.

The first decisive check is a controlled perturbation on ADNI. We apply randomly sampled intensity, bias-field and blur transforms either consistently to all of a patient's visits or to only one visit. The prediction is that the difference-first model is nearly unchanged under consistent transforms and degrades under single-visit transforms, while cross-sectional-first models degrade under both. If the difference-first model also degrades under consistent transforms, the invariance mechanism is not operating.

The second check is the scanner-change stratification within ADNI. The difference-first advantage should shrink on scanner-change pairs and partly recover when the flag is used. This pattern would show the mechanism working as claimed rather than added capacity.

The third check is gap decomposition. If the model's gain on external cohorts appears mostly in the population component rather than the scanner component, the invariance story is incomplete and must be revised.

The fourth check measures how often each model predicts a reversal of atrophy or diagnosis. It should be lower under domain shift for the difference-first model with the monotonicity loss than for the alignment baseline.