Brain-imaging models trained on one person transfer poorly to the next. Brains differ in folding and in where functions sit, so the same image activates different spots in different people. Large self-supervised models can learn these differences if they see enough people. The data, however, sits in hospitals and labs that cannot pool raw scans. Federated learning trains a shared model by averaging updates from each site, and it would sidestep the pooling problem. It creates a new problem, though: how much should each participant's update count? Suppose a participant's scans are hard for the shared model to fill in. That may be because their brain is organized unusually, in which case the model should learn more from them. Or it may be because they moved in the scanner, in which case the model should learn less. The literature gives opposite instructions for this case. One line of work up-weights hard participants. Another shows that doing so drives the model into fitting noise.
We propose to tell the two causes apart using the pretraining task itself. The model is a masked autoencoder, a network that hides most patches of an input and learns to reconstruct them. Each participant measures its reconstruction error on its own held-out scans at three levels of local adaptation. The first is the shared model as received. The second adds a tiny participant-specific input adapter. The third adds a brief, more expressive local fine-tune. Error that a tiny adapter removes is a shallow "neural accent" that can stay local. Error that only the expressive fine-tune removes is structure the shared model is missing, and that participant should get more weight. Error that nothing removes is a noise floor, and that participant should get less weight. The server receives three numbers per participant and sets aggregation weights from them. No labels and no clean validation set are needed.
Core Claim
In federated masked pretraining where each client is a single human subject, aggregation weights computed from a three-way split of each client's held-out reconstruction error will do two things. They will improve representations on unseen subjects relative to uniform averaging. They will also do better than both loss-up-weighting and loss-down-weighting rules, because they resolve the conflict between those rules instead of choosing a side. Let $L_g$, $L_a$ and $L_f$ be client $k$'s masked-reconstruction loss on its own held-out frames under the global model, the global model plus a shallow input adapter, and the global model plus a short low-rank fine-tune. Let $n_k$ be its number of frames. The server sets
$$ \alpha_k \propto n_k \exp\!\left(\beta\,\frac{L_a - L_f}{\tilde{m}_{af}} \;-\; \gamma\left(\frac{L_f}{\tilde{m}_{f}} - 1\right)\right), $$
where $\tilde{m}_{af}$ and $\tilde{m}_{f}$ are the cohort medians of $L_a - L_f$ and of $L_f$, and $\beta, \gamma \ge 0$ are two scalars. The shallow-accent term $L_g - L_a$ deliberately does not enter the weight. It is absorbed by the client's private adapter, which never leaves the site. In plain terms, the shared backbone listens hardest to subjects whose data it could learn from but has not yet learned, and it discounts subjects whose error no amount of local fitting removes.
The claim builds directly on a documented contradiction. Enhancing Cross-subject Emotion Recognition via Heterogeneous Distribution Augmentation and Collaborative Learning (MixEmo) gives higher weight to subject sub-distributions with higher query loss so that the model does not over-fit dominant subjects. From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching (FedISM) shows that this is a "fairness trap" when the loss comes from corrupted data, and replaces loss with a sharpness signal. When Less is More goes further and weights clients inversely to test loss. All three treat a client's loss as a single number. Our proposal treats it as a sum of three parts with different meanings, and the masking objective makes those parts measurable without labels.
Research Program
The data substrate is cortical flat maps: 3D cortical surfaces unfolded into 2D images so that a standard image transformer can process them. Scaling Vision Transformers for Functional MRI with Flat Maps (CortexMAE) introduced them and pretrained a masked autoencoder at a 90% masking ratio on about 2,058 hours from roughly 1,000 HCP-YA subjects. We reuse their representation and backbone recipe, which supplies the method. We do not use the flattened-voxel tokens of $i$MIND, because that paper itself lists the loss of spatial adjacency as a limitation. CortexMAE also reports that its centralized model degrades on out-of-distribution scanners and cohorts. That observation motivates our work: a model pretrained on one pooled cohort treats every scan identically and has no way to discount bad subjects.
We simulate a federation in which each HCP-YA subject is one client, so there are roughly a thousand clients with a few hours of data each. At each round a sampled subset of clients receives the global encoder and decoder and runs local masked-reconstruction training. Every few rounds, to control cost, each client also computes $L_g$, $L_a$ and $L_f$ on a held-out block of its own time series. The shallow adapter is a per-subject affine correction of the patch embedding, following the subject-adapter idea of See Through Their Minds. The difference is that we fit it self-supervised and keep it private. The expressive fine-tune uses low-rank adapters of the kind Rate-My-LoRA uses for federated cardiac segmentation. That paper scores updates against labeled local validation sets, whereas our scores come from reconstruction on unlabeled held-out frames. Both adaptation steps are capped at a fixed small number of iterations, so that $L_f$ estimates "what this client's data can teach in a short budget" rather than memorization. We will also check that $L_f$ is stable across a range of step counts.
The analysis has two parts. First, we validate that the three terms mean what we claim. $L_f$ should correlate with standard per-subject data-quality measures such as head-motion statistics and temporal signal-to-noise. $L_a - L_f$ should not correlate with them. We test this in a controlled setting by injecting synthetic corruptions into a known 10–30% of clients: motion-like spatial jitter, intensity drift, and additive noise. The 10–30% range mirrors the corruption ratios FedISM studies. Second, we test whether the weights improve the backbone on held-out subjects, compared against every competing rule under an identical training budget.
Baselines
The naive reference is federated averaging of the same masked autoencoder, with weights proportional to frame count. We also train the centralized CortexMAE-style model on the pooled data as an upper reference that ignores privacy. The strongest reported number for this setting is CortexMAE's own centralized result: 31.0% accuracy on NSD COCO24 and 98.9% on HCP-YA Task21 (Scaling Vision Transformers for Functional MRI with Flat Maps). We found no reported result on Lacuna for federated masked pretraining on fMRI, so every federated number in this study will be new. Beating federated averaging on held-out subjects, and closing a meaningful fraction of the gap to the centralized model, is the minimum bar. Matching or exceeding the centralized model under injected corruption would be the strong result. That outcome is plausible because the centralized model has no way to discount corrupted subjects.
The strongest existing adaptive-weighting competitor is FedISM, which pairs sharpness-aware local training with server weights set by each client's scalar sharpness. FedISM reports beating FedAvg by 10.85% and FedGA by 3.78% on corrupted RSNA ICH images (From Optimization to Generalization). That result comes from supervised CT classification, so it is not directly comparable to ours. We port FedISM to the masked objective and give it the same local compute. That matters, because its sharpness-aware step roughly doubles gradient cost, and our three-loss evaluation also costs extra. Our method should match or beat FedISM on held-out-subject probes and should down-weight injected-corruption clients at least as reliably.
The strongest self-supervised federated competitor is FedMAR. FedMAR adds a decaying distribution-alignment penalty to local masked training while keeping plain averaging on the server. It reports 70.82% CIFAR-100 accuracy with ResNet-18 against 38.44% for FedU (Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data). We run it as published, and also combined with our weights, because the two act on different stages and may stack.
We also run both directions of single-number loss weighting under identical budgets. One up-weights by global loss, in the style of MixEmo's weighted gradient update. The other down-weights by inverse loss, following When Less is More. Three ablations isolate our key ingredient. The first keeps only the $L_f$ penalty. The second keeps only the $L_a - L_f$ bonus. The third weights by $L_g - L_f$ without separating out the shallow accent. Finally, an oracle that zeroes clients known to be corrupted bounds what any quality-based rule could achieve.
Why not just…?
Why not just give every subject a private adapter and average the backbone uniformly? This is the most serious threat. Per-subject adapters, as in See Through Their Minds, may absorb most inter-subject variation by themselves, which would leave nothing for weighting to fix. Our design includes those adapters in every federated variant. The comparison of "adapters + uniform averaging" against "adapters + our weights" therefore answers the question directly. If the difference is within seed noise on clean data but appears under injected corruption, then the honest conclusion is that the weighting is a robustness mechanism, not an accent mechanism. We would report it that way.
Why not just weight subjects by standard quality-control metrics such as head motion or signal-to-noise? This could plausibly work as well for the noise half of the problem, and we test it as a baseline. Such metrics capture only the corruption types someone thought to measure. They also say nothing about which clean subjects carry structure the backbone lacks. If quality-metric weighting matches our full method, then the $L_a - L_f$ term adds nothing and the contribution reduces to a label-free proxy for quality control. The ablation that keeps only the bonus term settles this.
Why not just use FedISM's sharpness weighting, which already avoids the fairness trap? Sharpness flags clients whose loss surface is sharp. A sharp surface can come from noise, but it can also come from small, distinctive data. FedISM therefore discounts both, which is what the noise-versus-accent distinction is meant to prevent. Head-to-head under matched compute will show whether the extra discrimination matters. We will look most closely at clients that are clean but atypical, identified as subjects whose $L_a - L_f$ is high while their quality metrics are good.
Why not just pool the data centrally, as $i$MIND and CortexMAE do? Where pooling is allowed, it should be done, and we include it as a reference. $i$MIND and VCFlow both require all subjects together. VCFlow still reports a 7% average gap to subject-specific models. Pooling also treats every subject uniformly, so the weighting question exists there too. Our rule can run centrally as per-subject loss weights, and we will test that variant to separate the benefit of weighting from the constraints of federation.
Proposed Experiments
Pretraining uses HCP-YA flat maps with one subject per client, a fixed number of communication rounds, and an identical total local compute budget for every method. Evaluation freezes the encoder and trains linear probes on subjects never seen in pretraining. The probes cover the BRAINMARKS tasks reported by CortexMAE: HCP-YA Task21 on held-out HCP subjects and NSD COCO24. For NSD we hold out each of the eight subjects in turn, mirroring the leave-one-subject-out test of $i$MIND. We report mean accuracy, worst-subject accuracy, and spread across held-out subjects, since F²DC shows that spread reveals failures that averages hide. Every condition runs with clean data and with 10%, 20% and 30% corrupted clients, across at least three seeds.
Three checks decide the outcome. First, under corruption, the correlation between our weights and the true corrupted-client indicator should approach the oracle's, and it should exceed the correlation achieved by sharpness weighting. Second, on clean data, worst-subject held-out accuracy should improve over adapters with uniform averaging; if it does not, the accent half of the claim fails. Third, the single-term ablations should each lose to the full rule in at least one regime: the $L_f$-only rule on clean but atypical subjects, and the bonus-only rule under corruption. If both single-number loss rules, run in opposite directions, each fail in the regime the other handles, the split is empirically necessary.
We will also report the measured overhead of the extra local evaluations per round against FedISM's doubled gradient cost.
References
- Enhancing Cross-subject Emotion Recognition via Heterogeneous Distribution Augmentation and Collaborative Learning
- From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching
- When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning
- Scaling Vision Transformers for Functional MRI with Flat Maps
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding
- See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI
- Rate-My-LoRA: EFFICIENT AND ADAPTIVE FEDERATED MODEL TUNING FOR CARDIAC MRI SEGMENTATION
- Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data
- A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decoding
- Domain-Skewed Federated Learning with Feature Decoupling and Calibration