Cross-Modal Alignment Without Self-Matching: Removing the Brain Fingerprint from Multimodal Diagnostic Fusion

Many of the best brain-disease classifiers combine several scan types. A functional MRI connectome shows which brain regions co-activate over time; a diffusion MRI connectome shows the physical white-matter wiring between them. These models are often trained with a contrastive loss, an objective that pulls together the two scan types from the same patient and pushes apart scans from different patients. This proposal starts from a simple observation. "Make this person's functional scan match this person's structural scan, and no one else's" is, word for word, the training objective of a re-identification system. Separate work shows that combining MRI modalities makes individuals close to perfectly identifiable. The diagnostic models most celebrated for combining modalities may therefore be building the cross-modal brain fingerprint that privacy research warns about.

We propose to test this directly and to fix it at the level of the training objective. Our version of cross-modal contrastive alignment never rewards a subject's own pair. Positives are other subjects who share the diagnosis, and the subject's own pair is dropped from the loss. The two-sentence version is this: standard multimodal fusion teaches the model to recognize individuals across scan types. We change which pairs count as "matching" so that the model learns only structure–function agreement shared across patients, then measure how much diagnostic accuracy survives and how much re-identification disappears. A practical advantage is that the fix needs no identity labels or repeat scans. That matters because clinical cohorts usually have one session per person, so identity adversaries cannot be trained on them.

Core Claim

Our claim has three parts, each testable and each able to fail. First, instance-level cross-modal contrastive fusion measurably raises linkage between modalities. Given a subject's functional embedding, an attacker can retrieve that subject's structural embedding far above chance, and far above what a fusion model trained without the contrastive term allows. Second, redefining positives as same-diagnosis, different-subject pairs brings linkage close to chance and keeps most of the accuracy gain that contrastive fusion gives over concatenation. Third, doing this inside the objective yields a better privacy-versus-accuracy curve than scrubbing identity after training. If the third part fails, the objective change is unnecessary, and we will say so.

The loss is short enough to write down. Let $z_i^A$ and $z_j^B$ be unit-normalized embeddings of subject $i$'s modality-A scan and subject $j$'s modality-B scan, let $s_{ij}$ be their cosine similarity, $\tau$ a temperature, and $y_i$ the diagnosis label. The anchor-$i$ loss is

$$ \mathcal{L}_i = -\frac{1}{|P(i)|}\sum_{j\in P(i)} \log \frac{\exp(s_{ij}/\tau)}{\sum_{k\neq i}\exp(s_{ik}/\tau)}, \qquad P(i)=\{j\neq i : y_j = y_i\}. $$

Standard instance-level alignment corresponds to $P(i)=\{i\}$. The two changes are that $j=i$ is excluded from the positives and also from the denominator. The subject's own pair therefore gets no gradient in either direction, so the objective does not reward the model for recognizing that two scans came from the same person. The objective also does not penalize it, which keeps within-subject structure–function coupling available where it helps diagnosis. A mixing weight $\lambda\in[0,1]$ interpolating between this loss and the instance-level loss traces a privacy–utility curve. We also test a label-free variant in which $P(i)$ is the $k$ nearest other subjects in modality A. This checks whether diagnosis labels are needed or whether dropping self-matching is enough.

Relation to Prior Work

Two papers motivate the proposal by showing the leak exists. Multi-modal brain fingerprinting fuses T1w, T2w, diffusion and resting-state fMRI features on HCP. It reaches twin-identification mean average precision up to 0.997 and near-100% retest identification, with fusion beating every single modality. It treats this as a capability and offers no defense. De-anonymization Attacks on Neuroimaging Datasets re-identifies subjects from functional connectomes at over 94% on HCP rest and 97.2%/94.12% on ADHD-200 subtypes. Its authors state that a defense must strip signatures without damaging diagnostic biomarkers, and they leave that defense unbuilt. We build that defense for fused, not single, representations.

Three fusion architectures supply the method we modify. BrainNN aligns diffusion and functional graph embeddings with a global–local contrastive loss and credits it with about a 10% accuracy gain over concatenation on HIV. M2M-AlignNet aligns fMRI and sMRI patches contrastively before co-attention fusion. MIGTrans cascades attention across genomics, connectivity and sMRI. None of the three measures identifiability. We keep their encoders and change only which pairs count as positives. BrainNN's reported gain is also a risk to our claim: part of it may come from self-matching, and our experiments are designed to show how much.

Single-modality de-identification work supplies baselines and constraints. Mahalanobis whitening with Bures geometry drops subject-clustering adjusted mutual information to −0.00248 on healthy HCP fMRI while raising task clustering to 0.7516. It was never tested on patients, where disease may live in the very covariance it removes. Multi-IVE shows that removing embedding dimensions without protecting the leading components drives face-verification error to 34.5% EER, versus 13.6% with component locking. This warns that post-hoc removal can destroy utility. The EEG digital mask and Semi-Adversarial Networks show that adversarial generators work only with explicit task constraints; an unconstrained CycleGAN cut EEG identity accuracy to 0.48% but collapsed diagnosis. Those methods need identity labels and a generator, and our objective needs neither. pF-GMCO fuses sMRI with ALFF/ReHo across 23 sites under federated learning but does not examine what its representations leak. Our method sits inside such a pipeline without conflict.

Research Program

The first step is measurement. We train BrainNN-style graph encoders on paired diffusion and functional connectomes, and a lighter sMRI + ALFF/ReHo encoder in the style of pF-GMCO. We define two attacks, each run with a linear probe and with a two-layer MLP, because the face-privacy literature shows linear attackers understate leakage. The cross-modal linkage attack covers the case where modalities are released or stored separately. It ranks all subjects' B-embeddings for each A-embedding, reports top-1 accuracy and mean reciprocal rank against a chance level of 1/N, and needs only one session per subject, so it runs on every clinical dataset. The cross-session re-identification attack matches a subject's fused embedding from one HCP session to another session, including retest and rest-versus-task. It also includes the twin-matching protocol from the fingerprinting paper, so leakage is expressed on a scale with an established ceiling.

The second step is intervention. We train the same encoders with the self-excluded loss across a sweep of $\lambda$, plus the label-free neighbor variant, and plot diagnostic AUC against attack success for every method. The third step is diagnosis of the mechanism. We check whether linkage stays high after swapping the loss and, if it does, whether shared anatomy rather than the objective is responsible. This is the one scenario in which only representation-level scrubbing can help.

Baselines

The naive reference point is a fusion model with no privacy intervention, namely BrainNN's instance-level contrastive alignment. On HIV, BrainNN reports 77.14% accuracy and 79.79% AUC (BrainNN). We retrain it rather than copy numbers, so that our splits and attacks apply to it. Its linkage and re-identification scores define the leakage we aim to remove. On the utility side, the strongest reported figures are 86.05% accuracy on FBIRN from MIGTrans, which uses SNPs; we run an imaging-only reimplementation and compare to that rather than to 86.05%. The other figures are 79.07% accuracy on Rest-Meta-MDD from pF-GMCO and 71.55% ROC-AUC on EHBS from M2M-AlignNet, the latter only if data access allows. No reported result was found on Lacuna for cross-modal linkage or re-identification rates of any of these fusion models; producing those is part of the contribution.

The concatenation baseline, which is the same encoders with no contrastive term, isolates how much contrastive alignment contributes to accuracy and to leakage. The strongest privacy competitor is post-hoc erasure on the trained instance-level model. It comes in two forms: per-modality Mahalanobis whitening before fusion, following the whitening paper, and removal of the embedding directions most predictive of cross-modal linkage with the top components locked, following Multi-IVE. On HCP only, where repeat sessions provide identity labels, we add an adversarial identity branch trained with gradient reversal, in the spirit of semi-adversarial training. The key ablation keeps everything fixed and changes only $P(i)$. It compares self-only positives, self plus same-diagnosis positives with self kept in the denominator, and our self-excluded version, which isolates the effect of removing self-matching. All methods share encoders, data splits, a tuning budget of 30 random-search trials per method selected on validation AUC, and 5 seeds per configuration. Evaluation uses held-out sites where site labels exist. Beating the baselines means the following: at matched diagnostic AUC (within 1 point), our method's MLP-attacker linkage accuracy is lower than every baseline's, with non-overlapping seed intervals; or, at matched linkage, our AUC is higher.

Why not just…?

Why not just scrub identity from the trained embeddings afterward? This is the real threat. If a linkage-direction projection with component locking matches our curve, the objective change adds nothing. The face-embedding evidence argues against it: Multi-IVE's EER went from 13.6% to 34.5% when removal was unprotected, and nonlinear attackers there recover residual traits. That evidence is from faces, though, not brains. The head-to-head privacy–utility curve with MLP attackers settles it, and we will report the result whichever way it goes.

Why not just whiten each modality before fusing? Whitening erased identity on healthy HCP fMRI, but it removes the regional covariance in which connectivity-based disease markers are expressed. It was never evaluated on a patient cohort. It also acts on one modality at a time, so it cannot target identity that appears only in the agreement between modalities. Running whitened inputs through the same fusion pipeline on HIV, BP and Rest-Meta-MDD tests both concerns directly.

Why not just train an identity adversary on the fused embedding? It needs identity labels spanning multiple scans per person, which HIV, BP, FBIRN and most clinical cohorts lack. Where it can run, on HCP, the EEG ablation shows adversarial pressure without task constraints destroys diagnosis. We include it on HCP as a competitor. We expect it to reach similar privacy at some accuracy cost. If it beats us there, our method's advantage is limited to label-free applicability, and we will frame the result that way.

Why not just drop the contrastive term, or use one modality? Concatenation may leak less, but BrainNN attributes about 10 accuracy points to contrastive fusion. Single modalities trail fusion substantially: 74.55% for sMRI versus 79.07% fused on MDD. If our self-excluded loss keeps most of that gain, dropping alignment is dominated. If it keeps none, we will have shown that the "synergy" gain was largely subject self-matching, which is itself worth reporting.

Why not rely on federated learning? Federated training keeps raw scans on site but says nothing about what shared models and embeddings encode. The de-anonymization work found signatures survive multi-site noise above 90%. Our objective is complementary and can run inside a pF-GMCO-style client.

Proposed Experiments

HCP is the calibration set. There we pair diffusion and resting-state connectomes, report retest, cross-run and twin re-identification alongside cross-modal linkage, and use sex and age prediction as utility proxies. The clinical sets are HIV and BP for diffusion + fMRI graphs, FBIRN for FNC + sMRI (with SNPs added in a secondary run), and Rest-Meta-MDD for sMRI + ALFF/ReHo, where held-out-site evaluation tests whether identity leakage and site shift interact. The metrics are diagnostic AUC and accuracy, linkage top-1 and mean reciprocal rank against 1/N, re-identification accuracy and twin MAP on HCP, and area under the privacy–utility curve across $\lambda$.

Three checks decide the outcome. The first is whether instance-level fusion's linkage exceeds concatenation's; if it does not, the core premise fails. The second is whether the self-excluded loss keeps at least half of contrastive fusion's accuracy gain over concatenation while reducing MLP linkage to within a small margin of chance. The third is whether it beats post-hoc locked erasure on the full curve. A secondary check asks whether the label-free neighbor variant closes most of the gap. If it does, labels are not the source of the improvement, and the method applies to unlabeled pretraining.