Multi-hospital medical studies pool brain scans, blood or spinal-fluid assays, and cognitive test scores from many sites. Each site has its own scanner, lab, and test administrators, and these leave technical fingerprints in the data. The standard remedy is to "harmonize" each data type separately: fit a statistical correction to the MRI features, perhaps another to the lab values, then concatenate everything and train a classifier. We argue that this misses where the shortcut actually lives. A prediction model never sees one modality at a time. It sees the joint vector, and a site can stay identifiable from how its MRI features, lab values, and test scores move together even after each one has been individually corrected. The core idea fits in two sentences. Per-modality harmonization only matches each feature's site-specific mean and spread, so site identity can survive in cross-modal structure that a fused classifier is free to exploit. We propose to measure this leakage directly and to remove it with a harmonization model that is fitted on the joint feature space but that keeps each modality's statistical character separate and protects biological covariation by construction.
This matters because the field's evidence on whether multimodal fusion helps is mixed. Some ADNI studies show clear gains from adding MRI to cognitive and genetic features. Others find that extra modalities add nothing over clinical variables, or that adding spinal-fluid markers gives no significant improvement. One untested explanation is that the added modalities bring site noise in a form current pipelines do not remove. If joint leakage exists and is large, it explains part of that inconsistency. If it is negligible, the field can keep its simple pipelines with better justification than it has now. Either result is useful.
Core Claim
Our hypothesis is that, in multi-site cohorts, a classifier trained to predict acquisition site from the concatenation of separately harmonized modalities will do measurably better than one trained on any single harmonized modality. It should also do better than chance after biological covariates are regressed out. We call this excess the joint leakage gap. The gap should grow when clinical case mix differs across sites, and that is exactly the condition under which a fused disease classifier learns site as a proxy for diagnosis.
The proposed fix has three ingredients, each taken from existing work and recombined.
The first is biological protection by residualization, the step from DeepResBat. DeepResBat fits age, sex, diagnosis, and cognitive scores to MRI volumes with gradient-boosted trees, harmonizes only the residuals, and then adds the biological part back. That paper showed this keeps the false-positive rate at 5% under covariate permutation, whereas end-to-end conditional models invent associations. We reuse the residualization but drop its conditional autoencoder.
The second is empirical Bayes location and scale correction with modality-grouped priors. In ComBat (empirical Bayes batch correction), each feature's site-specific shift and scale are shrunk toward a distribution pooled across features. That pooling assumes features are exchangeable, which is false when cortical thicknesses, CSF tau, and word-recall scores sit in the same vector.
The third is a correction of the cross-modal covariance of the residuals, so that site-specific coupling between modalities is removed as well.
Concretely, for subject $j$ at site $i$ and feature $f$ in modality $m(f)$, we model the residual after biological adjustment as
$$ r_{ijf} = \gamma_{if} + \delta_{if}\,\varepsilon_{ijf}, \qquad \gamma_{if} \sim \mathcal{N}(\mu_{i,m(f)}, \tau^2_{i,m(f)}), \quad \delta^2_{if} \sim \text{InvGamma}(a_{i,m(f)}, b_{i,m(f)}). $$
Here $\gamma_{if}$ and $\delta_{if}$ are the site's additive shift and multiplicative scale for that feature, $\varepsilon_{ijf}$ is the remaining noise, and the hyperparameters $(\mu, \tau^2, a, b)$ are indexed by both site and modality. A site's MRI shifts therefore borrow strength only from other MRI features, and its assay shifts only from other assays. After this step, we estimate the site-specific joint covariance of the standardized residuals on a low-rank cross-modal basis. We replace each site's covariance with the pooled one, shrinking site-specific components when the site has few subjects.
Research Program
The work proceeds in three stages. The ordering matters: the method is only worth building if the first stage finds something.
The first stage is diagnostic. On ADNI, we harmonize MRI regional features, CSF markers, PET summaries, and cognitive scores per modality with ComBat-GAM, the additive-model variant used in the ENIGMA verbal memory mega-analysis. We then train site classifiers on each modality alone and on the concatenation, and compute the joint leakage gap. We do this both on raw harmonized features and on biology-residualized features, so that legitimate case-mix differences are not counted as leakage. The motivating evidence is indirect. Batch Effects In Brain Foundation Model Embeddings shows that frozen fMRI embeddings predict site at 85–94% accuracy against 66–71% for diagnosis. It also shows that ComBat lowers recoverable site information without making biology any clearer. That paper studied one modality. We ask whether the same failure appears across modalities.
The second stage builds the joint model described above and compares it with the alternatives under a common protocol.
The third stage addresses unseen sites. Because the grouped priors give a predictive distribution over a new site's parameters, a new hospital can be harmonized from a small reference sample, even one that lacks a modality, by drawing on its modality-level hyperparameters. We compare this against Cluster ComBat, which clusters site parameters with K-means so that unseen MRI sites can be harmonized. That paper reports lower error on unseen sites than standard ComBat but, as summarized, gives no exact figure. It is also unimodal and leaves the choice of cluster count open.
The work is positioned against three neighbors. RegBN is the closest. It is a normalization layer that linearly projects one modality's features out of another before fusion, which removes shared dependence. It has no site parameters, and it treats all cross-modal dependence as nuisance. We take the opposite stance: cross-modal covariance is often biology, as with atrophy tracking tau, so we remove only its site-specific part. InVA uses "harmonization" to mean predicting amyloid PET from MRI through a shared latent space. It ignores scanners, so it supplies no batch mechanism. The MS subtyping study applied ComBat to MRI volumes but not to serum neurofilament, despite known assay variability. That asymmetric, siloed practice is exactly what our diagnostic stage evaluates.
Baselines
The naive reference point is no harmonization. Raw multimodal features are concatenated into Lasso and XGBoost classifiers. Next to it sits a clinical-only model that uses cognitive scores, APOE genotype, and demographics. This anchors whether any imaging or fluid modality is worth its noise.
The strongest practical competitor is per-modality ComBat-GAM followed by concatenation. It is what careful applied studies already do, for example the CHR psychosis prediction pipeline, which used ComBat, then additive-model age and sex adjustment, then XGBoost. We will also run a per-modality variant with DeepResBat-style residualization and ComBat, so that our method's advantage cannot come from residualization alone. The best ADNI MCI-conversion number in the file is AUC 0.8587 for the 15-feature sparse multimodal biosignature in Sparse learning and stability selection. DTSVM reports AUC 0.736 on a smaller MRI+PET+CSF cohort. Both used non-site-held-out validation, so we will rerun their feature sets under our splits instead of quoting them as targets. For RegBN on ADNI dementia diagnosis, the file reports 53.0% accuracy. For DeepResBat's site-prediction accuracy, no reported number was found on Lacuna.
Two ablations isolate the key ingredients. The first applies a single ComBat to the concatenated vector with one shared prior, which tests whether the grouped priors matter. The second runs our model without the cross-modal covariance step, which tests whether the leakage lives in covariance rather than in per-feature location and scale. RegBN will be applied to modality encoders on the same tabular features. The unseen-site comparison uses Cluster ComBat.
All methods get the same splits, the same downstream classifiers, and the same hyperparameter search budget, measured as an equal number of configurations tuned on inner folds. Splits are leave-site-out folds within ADNI, plus ADNI-to-AIBL transfer on the overlapping modalities.
Our method counts as beating per-modality ComBat-GAM only if three things hold together. It must close the joint leakage gap, meaning residualized site balanced accuracy is statistically indistinguishable from chance where the baseline is not. It must give equal or better leave-site-out conversion AUC, with a gain exceeding the bootstrap confidence interval under case-mix-confounded splits. And it must keep a nominal 5% false-positive rate under permutation.
Why not just…?
Why not just run ComBat on each modality and concatenate? This is the real threat, and stage one exists to settle it. If the joint leakage gap is near zero on residualized features, the proposal reduces to a negative but useful result: siloed harmonization suffices for tabular multimodal fusion, and the MS study's asymmetric practice is the only thing to fix. Our reason to expect a gap is mechanistic. ComBat matches only per-feature means and variances. The foundation-model results show that matching these moments leaves site information intact in a structured representation.
Why not just run one ComBat on the concatenated vector? It is cheap and might work. But its pooled prior shrinks CSF assay shifts toward MRI thickness shifts, which have unrelated scales and noise. With few subjects per site, that pulls small-modality estimates toward the wrong target. The single-prior ablation tests this directly. If it matches the grouped model, we adopt the simpler version.
Why not just drop the noisy modalities and use clinical data? Sometimes that is right; the head and neck cancer study found clinical features alone were strongest. On ADNI, though, the sparse-learning study reports AUC 0.8111 for cognitive, genetic, and demographic features against 0.8587 when MRI is included. The clinical-only baseline tells us, under site-held-out evaluation, whether that margin survives.
Why not just train an adversarial or conditional autoencoder end to end? DeepResBat's permutation experiments show that end-to-end conditional models report biological effects even when covariates are scrambled. InVA also reports that plain autoencoder baselines predict worse than a tree ensemble (MSPE 0.08–0.10 versus 0.0667). With a few hundred subjects, a parametric shrinkage model is the more defensible starting point. We include a domain-adversarial baseline in the permutation test to confirm this on multimodal inputs.
Proposed Experiments
The decisive checks are four. The first measures the joint leakage gap on ADNI, with permutation-based significance on the difference between joint and best single-modality site accuracy. The second is a case-mix stress test modeled on the confounded design in the foundation-model paper. We subsample sites so that converters concentrate at some sites and non-converters at others, then measure how far each pipeline's leave-site-out AUC falls relative to balanced splits. This is where joint leakage should hurt most.
The third check guards against hallucination and over-correction, and it is the main risk of the covariance step. We run covariate permutation tests on the cross-modal associations (MRI atrophy versus CSF tau) and require a 5% false-positive rate. Then, in a semi-synthetic setting, we inject a known biological cross-modal correlation and a known site-specific coupling, and verify that the method removes the second while recovering the first. The fourth check is unseen-site harmonization from 5, 10, and 20 reference subjects, including sites with a missing modality, compared with Cluster ComBat.
If the injected biological correlation is attenuated beyond its confidence interval, the covariance step is removing biology, and we would fall back to location and scale correction only.
References
- DeepResBat: deep residual batch harmonization accounting for covariate distribution differences
- Verbal Learning and Memory Deficits across Neurological and Neuropsychiatric Disorders: Insights from an ENIGMA Mega Analysis
- Batch Effects In Brain Foundation Model Embeddings
- Distributed Harmonization: Federated Clustered Batch Effect Adjustment and Generalization
- RegBN: Batch Normalization of Multimodal Data with Regularization
- InVA: Integrative Variational Autoencoder for Harmonization of Multi-modal Neuroimaging Data
- Combined magnetic resonance imaging and serum analysis reveals distinct multiple sclerosis types
- Using brain structural neuroimaging measures to predict psychosis onset for individuals at clinical high-risk
- Sparse learning and stability selection for predicting MCI to AD conversion using baseline ADNI data
- Domain Transfer Learning for MCI Conversion Prediction
- Machine learning-based treatment outcome prediction in head and neck cancer using integrated noninvasive diagnostics