Is Head Motion the Hidden Site Fingerprint in Brain Foundation Models?

Large neural networks pretrained on brain scans are supposed to give researchers a shared representation of brain activity that works across hospitals and studies. In practice, they mostly learn which scanner a scan came from. On pooled resting-state functional MRI (fMRI, which measures blood-oxygen fluctuations over time), embeddings from two such models identify the acquisition site with 85–94% accuracy but predict diagnosis at only 66–71%. Statistical harmonization applied afterward removes the site label from the embeddings without making the biology any clearer (Batch Effects In Brain Foundation Model Embeddings). We propose that a large part of this "site" signal is something physical and measurable: how each scanner's images respond to the patient's head moving. Every fMRI run records a frame-by-frame head-position trace. That trace lets us estimate each site's motion response from fluctuations within a single scan, where diagnosis is held fixed, and subtract it before data are pooled.

The core idea in two sentences: instead of regressing out a subject's average motion (which also removes disease signal, because patients move more), learn per site how the embedding of a short time window shifts as head motion in that window changes, using only within-subject variation, and remove that shift. The test is whether this removes site fingerprints that ordinary harmonization leaves behind, and whether diagnostic effects then agree across independent consortia.

Core Claim

Integrated motion correction treats motion as a transformation to model and invert, not as data to throw away. The motion-correction literature shows clearly that modeling motion beats discarding it. Warping all acquired PET events with MRI-derived motion fields cut noise by 71.8–95.2% relative to gating (MR-based motion correction for cardiac PET parametric imaging). Rebinning raw events recovers structure that post-hoc image warping cannot (MRI-Assisted PET Motion Correction). These papers motivate the principle, but we cannot reuse their raw-data machinery on retrospective consortium fMRI, where only reconstructed images and realignment parameters exist.

We carry the principle into the representation space where multi-site analyses now happen. Our claim is that the site-by-motion interaction is a separable, identifiable part of the batch effect, and that removing it fixes what scalar harmonization cannot. Scalar harmonization here means ComBat-style empirical-Bayes adjustment of per-site location and scale (Radua et al.). It cannot handle this component because the component is nonlinear, site-specific, and correlated with diagnosis. Gong et al. explicitly flag that harmonization ignores geometric effects, and this proposal targets that gap. Jensen et al. note that unmodeled patient–control motion differences contaminate cross-consortium resting-state findings. That note motivates the meta-analytic payoff.

Research Program

Take a frozen foundation model, either BrainLM (a masked-autoencoder over parcellated time series) or SwiFT (a 4D Swin transformer on volumes). Embed each subject's run as overlapping short windows, so each subject contributes many windows. For window $w$ of subject $i$ at site $s$, let $z_{iw}$ be the embedding and $m_{iw}$ a motion descriptor built from the six realignment parameters. The descriptor includes framewise displacement, peak displacement, rotation and translation magnitudes, and the previous window's values, so that lasting spin-history effects are captured. We fit

$$ z_{iw} = \alpha_i + f_s(m_{iw}) + \varepsilon_{iw}, $$

where $\alpha_i$ is a per-subject fixed effect that absorbs diagnosis, age, sex and every other stable trait, and $f_s$ is a small site-specific function: a spline or a two-layer network, shrunk toward a pooled $f$ by empirical-Bayes-style partial pooling, as ComBat does. Because $\alpha_i$ soaks up everything constant within a subject, $f_s$ is identified only from moment-to-moment motion fluctuations. This means diagnosis cannot leak into the motion model even though patients move more on average.

The corrected window embedding is $z_{iw} - f_s(m_{iw}) + f_s(0)$, the embedding projected to zero motion. Windows are then averaged into subject embeddings, and ordinary ComBat is optionally applied to whatever site offset remains. No frames are discarded.

The analysis has three parts. First is a decomposition: what fraction of the site variance in the embeddings (PERMANOVA pseudo-F, linear site decodability) is accounted for by $f_s$ versus residual scanner offsets? Second is correction: does removing $f_s$ improve held-out-site diagnosis and biological decoding? Third is meta-analysis: do patient–control effect sizes computed separately per consortium become more consistent once motion response is removed?

The central assumption is that the within-subject motion response transfers to between-subject motion differences. We check it directly in subjects whose motion varies widely across a run, by comparing their high-motion and low-motion halves.

Baselines

The naive reference is raw frozen embeddings with no harmonization. Tao et al. report 85–94% site accuracy and 66–71% diagnosis accuracy for this condition, and on FBIRN a BrainLM site pseudo-F of 717.35 against a diagnosis pseudo-F of 0.15.

The strongest existing method is ComBat applied to the frozen embeddings, as run by Tao et al. After ComBat, BrainLM decodes regional ALFF (a voxelwise measure of low-frequency signal amplitude) with $R^2 = 0.115$. It is the strongest competitor because it is what the field does now and because it already drives site decodability down. Beating it means three things together: matching or lowering its site decodability, improving held-out-site diagnosis AUC, and raising ALFF $R^2$ above 0.115 on the same BrainLM embeddings. The third matters because Tao et al.'s point was that site removal alone did not clarify biology.

A third baseline is ComBat with subject mean framewise displacement as a covariate. This is the obvious "motion-aware" variant and is discussed below. A fourth is scrubbing, which drops frames and subjects above framewise displacement thresholds of 0.2 and 0.5 mm, followed by ComBat. For the pooling question we compare against random-effects meta-analysis of per-site effects. Radua et al. report its power at 53.7% versus 83.5% for ComBat-Mega on ENIGMA structural data. No reported result was found on Lacuna for random-effects meta-analysis versus mega-analysis on these fMRI benchmarks.

Two ablations isolate the key ingredient. The first fits $f_s$ from between-subject variation, without $\alpha_i$; this tests whether within-subject identification matters. The second replaces site-specific $f_s$ with one pooled $f$; this tests whether the site-by-motion interaction, not motion alone, carries the fingerprint.

For fairness, every method sees the same subjects, the same windows, the same frozen backbone, and the same linear probe. Every tunable choice (spline knots, network width, scrubbing threshold, ComBat covariate set) gets the same hyperparameter search budget, chosen by cross-validation on training sites only.

Why not just…?

Why not just add mean framewise displacement as a covariate in ComBat? This is the real threat, and it may work for part of the effect. It has two weaknesses we can test. Because motion differs systematically between patients and controls (Jensen et al.), a between-subject motion covariate partly absorbs diagnosis. It also assumes one linear motion slope shared across scanners. Our between-subject ablation is essentially this method in a flexible form. If it matches the within-subject version on diagnosis AUC and ALFF $R^2$, we will report that the simple fix suffices and that the site-by-motion component adds nothing. We consider that outcome possible.

Why not just scrub high-motion frames and subjects? Discarding data is the representation-space equivalent of gating, which the PET literature shows inflates variance (Guo et al.). It also selectively removes patients, biasing the comparison. We will report how many subjects per diagnosis each threshold drops, alongside accuracy, so any apparent gain from scrubbing can be checked against the sample it was computed on.

Why not just pool per-site effect sizes with random-effects meta-analysis and skip harmonization? Random-effects meta-analysis lost substantial power relative to mega-analysis in ENIGMA (Radua et al.). More importantly, it does not remove a motion confound that pushes every site's patient–control effect in the same direction. Averaging consistent bias yields confident, wrong results. We use it as the pooling comparison and expect it to stay biased, not merely underpowered.

Proposed Experiments

We use the multi-site resting-state benchmarks from Tao et al.: FBIRN (schizophrenia), ADHD-200, and ABIDE I (autism). Realignment parameters come from a single shared preprocessing pass.

The first decisive check is decomposition. If $f_s$ explains a negligible share of site pseudo-F, the hypothesis is false and we report that as the main finding.

The second is Tao et al.'s confounded design, where all patients come from one site and all controls from another and a probe reaches 100% false accuracy. If the shortcut is partly motion-mediated, the false accuracy should fall after our correction more than after ComBat. Diagnosis accuracy in unconfounded leave-one-site-out testing should hold or rise.

The third is cross-consortium consistency. We compute schizophrenia-versus-control effects on the circuits Jensen et al. found reproducible (cerebellar–thalamic hypoconnectivity, cortical–subcortical hyperconnectivity), using connectivity read out from corrected embeddings, and measure between-site heterogeneity. Heterogeneity should drop. The prefrontal effects Jensen et al. found unreliable should shrink if they were motion artifacts, not grow.

As an external sanity check, the single-site prospective-correction cohort of Lappin et al. offers runs with tracking on and off. The estimated $f$ should be measurably flatter when prospective correction is on, which would confirm that $f$ captures a physical motion response and not an arbitrary nuisance direction.