Recent brain foundation models learn from raw 3D brain volumes over time, the full voxel grid, instead of from averages over a few hundred predefined brain regions. On benchmarks drawn from the same studies they were tested within, they do better. This proposal asks whether that gain survives when the test cohort comes from a site, scanner, and population the model never saw. The reason to doubt it is concrete. The details that voxel models keep and region averages throw away, such as absolute signal intensity and fine spatial texture, are the same details that differ systematically between scanners and acquisition protocols.
We will run the first controlled test of frozen voxel-level fMRI embeddings under leave-one-dataset-out evaluation. A frozen embedding is a fixed feature vector from a pre-trained encoder whose weights are not updated, with only a small linear predictor trained on top. We will measure how much site identity those embeddings carry. We will then test one specific explanation and one fix. The explanation is that keeping raw intensity, which helped within-dataset accuracy, also preserves scanner gain. The fix is an intensity normalization at the level of each scan session, which keeps contrast between regions but removes the global scale. If the explanation holds, it changes how future volumetric neuroimaging encoders should be pre-trained.
Core Claim
Omni-fMRI pre-trains a 157M-parameter Vision Transformer on 4D fMRI voxel grids from 49,497 sessions. Two of its findings matter here. First, linear probes on its frozen features outperform fully fine-tuned baseline foundation models across 16 tasks. Second, the standard per-patch normalization ("Patch-Norm") hurts because it destroys absolute BOLD intensity. BOLD is the blood-oxygen signal that fMRI measures.
Its evaluations are pooled or within-dataset, though. In a separate line of work, Detect and Correct Bias in Multi-Site Neuroimaging Datasets shows that a random forest identifies which of 17 studies a scan came from with 71.5% accuracy using only 125 FreeSurfer summary measures. Raw voxel intensities should carry at least as much scanner information as those summaries.
Our claim is that part of the voxel-over-atlas advantage is site leakage carried through absolute intensity. If so, three things should follow. Frozen voxel embeddings should identify the source site more accurately than region-level embeddings. Their advantage should shrink or reverse under leave-one-dataset-out evaluation once age ranges are matched. And normalizing global intensity per session should recover most of the within-dataset benefit of skipping Patch-Norm while cutting site leakage.
The other directions play supporting roles. Volumetric context is the object of study. Frozen feature extraction is the deployment regime, chosen because it is what Omni-fMRI and Brain-JEPA recommend. That regime also cannot repair a biased feature space on its own, because the frozen weights never adapt to the new site. Population bias is the evaluation lens. Multimodal fusion enters as one secondary test, described below, of whether adding a structural modality dilutes site signal or adds more of it.
Research Program
The evaluation bed is the six-dataset rs-fMRI benchmark of Benchmarking External Generalization of SPD Matrix Learning: COBRE, ADNIDOD, Cam-CAN, ABIDE, OASIS-3 and ADNI, with brain-age prediction under leave-one-dataset-out (LODO) evaluation. That paper reduced each scan to a correlation matrix between regions and trained small models from scratch. We keep its splits but use raw 4D volumes and frozen pre-trained encoders.
One confound has to be removed first. That paper reports MAE above 30 years when ABIDE, the youngest cohort, is held out, because no training dataset contains young participants. That is age-range extrapolation, not site shift. We will therefore report two LODO variants: the original one, and one in which each held-out cohort is restricted to the age range covered by the training cohorts.
We compare three families of frozen encoders. The first is voxel-level: Omni-fMRI's released weights if available, otherwise our reimplementation. The second is region-level with gradient positions: Brain-JEPA, which uses 450 regions plus coordinates taken from the main axes of brain-wide connectivity patterns, using its official code. The third is region-level multimodal: Brain-OF with its 246-region fMRI branch.
Pre-training our own voxel variants is affordable, since Omni-fMRI reports 32 hours on four A10G GPUs. We will train three: Patch-Norm, no normalization (the Omni-fMRI default), and session-level normalization. The session-level version rescales each session by its global brain-masked mean and spread, so differences between regions survive but scanner gain does not. All three see the identical pre-training corpus with the same masked-reconstruction objective. A fourth variant swaps reconstruction for latent prediction, which lets us also probe the reconstruction-versus-latent-prediction (MAE versus JEPA) dispute between the two papers specifically for site robustness. Every downstream cohort will be checked for subject overlap with the pre-training corpus.
For each encoder we measure three things. Site leakage is the accuracy of a linear probe predicting source dataset from the frozen embedding, with age and sex regressed out first so that site cannot be inferred from demographics. Within-dataset brain-age MAE comes from cross-validation inside each dataset. The LODO gap is LODO MAE minus within-dataset MAE, and R² is reported alongside it. The central analysis is whether site leakage predicts the LODO gap across encoders and normalization variants.
We also add a cheap correction at the probe level, which we call target-conditioned site nulling. On the training datasets we find the embedding directions that predict site after the age signal has been regressed out, and project them away before fitting the age probe. Unlike unsupervised harmonization, this is designed to leave age-related variance in place. That matters because of the double-edged effect Wachinger et al. describe: harmonization can strip biology as well as site.
The secondary fusion test concatenates frozen fMRI embeddings with FreeSurfer morphometrics in ADNI and OASIS-3, both of which have T1 structural scans. We then ask whether fusion improves LODO error or mainly raises site leakage. We cannot test EEG–fMRI fusion under LODO. Brain-OF's reported fMRI-to-fusion gain on LEMON, from 61.79% to 81.10% brain-age accuracy, comes from a single cohort, and no multi-site paired EEG–fMRI benchmark exists in the material. We say this openly rather than overclaim.
Baselines
The naive reference point is a ridge regression of age on mean signal intensity per region plus head-motion summaries, trained under the same LODO splits. If frozen foundation features cannot beat it on the held-out cohorts, they are not adding transferable biology. Alongside it we run the Tangent-Space Ridge connectome model from the SPD benchmark. That paper found this model's advantage over deeper models persisted under LODO, and it reports MAE above 30 years and frequently negative R² in some LODO cases. We will rerun it on our exact splits, including the age-matched variant, instead of copying those numbers.
The strongest existing method is a linear probe on frozen Omni-fMRI features. Its paper reports that such probes beat fully fine-tuned NeuroSTORM and SwiFT, but no reported result was found on Lacuna for frozen voxel-level encoders under LODO. We will establish that number ourselves. Brain-JEPA is the strongest region-level competitor because it is the only encoder in the file explicitly evaluated for cross-ethnic transfer, on NC versus MCI in the Asian MACC cohort. No numerical result for that transfer is given in the file. All probes will be fit with the same nested hyperparameter search on training datasets only, use identical preprocessing and embedding dimensionality budgets, and never see the held-out cohort's labels or statistics.
The ablation that isolates our key ingredient is the normalization sweep over Patch-Norm, none, and session-level, with the pre-training corpus, architecture, and compute held fixed. Beating the baselines means the session-normalized encoder plus nulling achieves lower age-matched LODO MAE than frozen Omni-fMRI-style features on at least five of six held-out cohorts, with within-dataset MAE no more than modestly worse. Site-leakage accuracy should also drop toward the levels seen in region-level encoders.
Why not just…?
Why not just apply ComBat++ to the frozen embeddings? This is the most serious threat. Applied to 17 datasets of morphometric features, ComBat++ reduced dataset-prediction accuracy from 69% to 45% while strengthening the expected age–hippocampus relationship (Wachinger et al.). Two caveats limit it. Forty-five percent is still far above chance. And the SPD benchmark found Riemannian harmonization helped linear models but gave inconsistent gains under LODO for deeper ones. ComBat++ on embeddings will be run as a full competitor. If it matches session normalization plus nulling, we will report that the fix is post hoc and cheap. The diagnostic finding that intensity carries site signal would still stand.
Why not just use region-level encoders like Brain-JEPA, which already transfer across ethnic groups? Averaging within atlas regions may well suppress scanner texture. Omni-fMRI, however, argues that atlases impose group-template bias and break when cohorts are registered to different templates. So the honest question is which costs more out of distribution, atlas bias or site leakage. Our three-family comparison under age-matched LODO answers this directly. If Brain-JEPA wins, that is a publishable negative result for voxel modeling.
Why not just fine-tune end-to-end with a site-adversarial loss? Fine-tuning a 157M-parameter encoder on a few hundred subjects per site risks the small-sample overfitting that motivates frozen probing. It also requires retraining for every new deployment. We include an adversarially fine-tuned voxel encoder, given the same tuning budget as the probes, as an upper-cost reference. If it clearly dominates, the frozen-regime recommendation from Omni-fMRI and Brain-JEPA needs qualifying for multi-site use.
Decisive Checks
The hypothesis fails if two things are both true: frozen voxel embeddings carry no more site information than region-level ones after demographic control, and the normalization sweep leaves the LODO gap unchanged. In that case the voxel advantage is real biology, and the proposal's warning is unfounded.
It succeeds if three things hold. Site leakage should track the LODO gap across the encoders. Removing Patch-Norm should raise both within-dataset accuracy and site leakage. And session-level normalization should keep most of the first effect while removing most of the second. The age-matched LODO split is what makes either outcome interpretable, because without it ABIDE's age-range failure would swamp any site effect.
References
- Omni-fMRI: A Universal Atlas-Free fMRI Foundation Model
- Detect and correct bias in multi-site neuroimaging datasets
- Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking
- Benchmarking External Generalization of SPD Matrix Learning for Resting-State fMRI Connectome Prediction
- Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG