Brain Decoders That Adapt to a New Person From One Scan Session

Models that reconstruct or identify what a person is seeing from their functional MRI (fMRI) brain scans work well only when trained on many hours of that one person's data. A new participant does not get a model for free, because every brain is laid out a little differently. Current multi-person decoders either train a fresh input layer for each newcomer or align the newcomer's voxels to a reference person, and both need hours of scanning plus gradient training. We propose a different route, taken from speech synthesis. There, a model imitates an unseen speaker's voice from a few seconds of audio by having a short reference clip set the scale and shift inside every normalization layer. This is conditional layer normalization: the per-feature rescaling that ordinarily follows normalization is predicted from a context signal, here the new person's short calibration scan, rather than learned once and frozen.

Moving this trick to brain data runs into a sample-size problem one level up. Speech systems learn the mapping from reference clip to scale and shift from thousands of speakers. The standard fMRI benchmark has 8 people, and most decoding work uses only 4 of them. A predictor of normalization parameters trained on so few people should overfit to who those people are. The central idea of this proposal is to train the conditioning on scan sessions rather than on people. Each Natural Scenes Dataset participant contributes dozens of sessions, so the conditioner sees hundreds of training "contexts" and has to learn what varies across sessions and across brains. We test whether that is enough for a decoder to adapt to a held-out person with no gradient steps, using one session or less of calibration data.

Core Claim

The claim has three parts. First, a shared decoder works in a common cortical surface space, so a new person needs no new per-voxel input layer. Second, inside that decoder, every layer normalization has its scale and shift predicted from a compact context vector. Third, that vector is inferred in a single forward pass from a short calibration set of (image, brain response) pairs, and it is constrained to be a mixture over a small learned dictionary of "basis contexts."

Our prediction is that this matches or beats gradient-based per-subject calibration once calibration data is scarce, roughly 50 to 500 trials. We expect the advantage to come mostly from the low-dimensional, session-trained parameterization. Feed-forward inference adds speed but should add little accuracy. The core question is about sample size: does conditioning learned from fewer than 8 brains transfer to a ninth? Every main result is therefore reported under leave-one-subject-out evaluation.

Research Program

The mechanism comes from two speech papers. AdaSpeech 4 puts conditional layer normalization in both the encoder and decoder of a text-to-speech model. It derives the conditioning vector by letting reference audio attend over 2,000 learned speaker basis vectors, and removing that basis regularization cost 0.46 CMOS (a listener preference score). Adapter-Based Extension of Multi-Speaker TTS conditions a frozen backbone through the same kind of normalization while tuning about 7% of parameters on 5 minutes of audio. We reuse the basis-mixture idea. A dictionary of 2,000 entries is clearly too large for our setting, so dictionary size becomes an experimental variable, from 8 to 128.

The decoder input is each trial's single-trial response estimate, computed with GLMsingle and sampled on the shared fsaverage surface. A shared linear patch embedding feeds a transformer of about the size of VoxelFormer. The transformer outputs CLIP image embeddings and is trained with the contrastive and regression objectives used by VoxelFormer.

The context encoder is a small set-pooling network. It takes $M$ calibration pairs, each the CLIP embedding of an image plus the surface response to it, and returns attention weights over $K$ basis vectors. The weighted sum gives the context vector $e$. Linear maps of $e$ then produce the scale and shift for every layer norm.

During training, each minibatch draws calibration trials from one session and decoding targets from a different session of the same person. This forces $e$ to capture the stable signature of a brain rather than within-session noise. We also add a variance penalty that keeps the context vectors of the same person's sessions close together, which plays the role of AdaSpeech 4's distribution supervision.

The analysis side asks what the inferred contexts encode. We check whether they cluster by subject, whether they track session-level signal quality, and how they change as $M$ shrinks. The last point bears directly on the concern that dynamic normalization overfits to idiosyncrasies of a short reference, raised in Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction.

Baselines

The naive reference point is the same shared surface-space transformer with no subject information at all, applied unchanged to the held-out person. This tests whether a common anatomical space plus a shared network already absorbs individual differences. VoxelFormer argues that a fully shared model is viable, reporting 74.3% mean forward top-1 retrieval out of 300 candidates. That figure was measured on subjects seen in training, not held-out ones. Beating this baseline means a reliable gain in held-out-subject retrieval at every calibration budget.

The strongest existing method for the calibration-budget question is ridge-regression functional alignment from Through their eyes. It maps a new person's voxels into a reference person's space and feeds them to that person's frozen decoder, and it beats Procrustes hyperalignment. Using 982 shared images, about 10% of scan time, it gave reconstructions comparable to full-data training. No reported result found on Lacuna for its top-1 retrieval accuracy. We will therefore rerun it ourselves on the same held-out subjects, fitting ridge on exactly the same $M$ calibration trials, choosing the regularization by nested cross-validation within those trials, and decoding through our trained shared model rather than a different pipeline. Beating it means higher retrieval at $M \le 500$. At $M = 982$ we expect parity at most.

The token baseline follows MindFormer, which prepends a learned subject token and keeps semantic fidelity down to 500 training samples per subject. We train it on the same shared model and fit the held-out person's token by gradient descent on the calibration set. This is the direct test of whether conditioning at every layer outperforms conditioning only at the input.

The gradient-calibration baseline is BitFit-style tuning of the static layer-norm scales and shifts, the parameter-efficient method compared in the adapter paper. It gets the same calibration data and an equal tuning budget, with early stopping on held-out calibration folds.

The ceiling is a per-subject model trained on all of that person's data. MindEye1 reaches about 93% top-1 retrieval in this setting, as reported in the VoxelFormer paper.

Two ablations isolate the key ingredient. The first fits only the $K$ mixture weights by gradient descent, with no context encoder. It separates the value of the low-dimensional basis from the value of amortized feed-forward inference. The second trains the conditioner on subject-level rather than session-level contexts, which tests the sample-size argument at the heart of the proposal.

Why not just…?

Why not just align the new person's voxels with ridge regression? It is cheap and it works: Ferrante and colleagues show it beats hyperalignment and nearly matches full-data decoding at 982 images. Its weakness is that it estimates a voxel-by-voxel map, tens of thousands of inputs by tens of thousands of outputs, from the calibration set. We therefore expect it to degrade faster than a $K$-dimensional context as $M$ falls. If it matches our method even at 100 trials, then normalization-based conditioning adds nothing, and the calibration-budget curve in our experiments will show that directly.

Why not just prepend a subject token? MindFormer's results show tokens work when the subject is seen in training. The washout argument from AdaSpeech 4 comes from speech, not fMRI, so we do not assume it holds here. We test it with three variants at equal parameter count: a token only, conditional normalization in the first block only, and conditional normalization in all blocks. If the token matches the all-blocks variant, the claim about layer-wise conditioning fails. The session-trained basis might still help, in which case it should be attached to the token instead.

Why not just fine-tune the layer-norm parameters for each new person? This is the most serious threat. A learned per-person embedding pushed through a linear map is just a low-rank constraint on per-person layer-norm parameters. Our only real advantages are the low rank, shared structure learned across sessions, and skipping test-time optimization. The mixture-weights-by-gradient ablation is built to settle this. If it equals the full method, we will report that the gain comes from the basis, not from feed-forward inference. If plain BitFit equals both, the proposal reduces to "fine-tune layer norms," and we will say so.

Why not just use a fully shared model? It needs no calibration, but VoxelFormer's own numbers leave about a 19-point gap to specialized models even on seen subjects. The naive baseline measures how much larger that gap is on unseen ones.

Proposed Experiments

All decoding experiments use the Natural Scenes Dataset with leave-one-subject-out evaluation over its subjects. The metric is top-1 forward and backward retrieval among 300 candidates on the shared test images. We plot retrieval against calibration budget $M \in \{25, 50, 100, 250, 500, 982\}$, drawing calibration trials from a single session. Every run is repeated over calibration draws and held-out subjects, with confidence intervals across subjects, since subjects are the unit that limits sample size.

The first decisive check is whether the gap between our method and both ridge alignment and BitFit widens as $M$ shrinks. That shape is the signature of a better-constrained parameterization. A constant offset would point instead to architecture differences.

The second check compares conditioners trained on sessions and on subjects. If session-level training does not improve held-out-subject accuracy, the proposal's answer to the eight-subject problem is wrong.

The third check is a shuffled-context control. Feeding a different person's calibration set should hurt accuracy; if it does not, the context carries no information about the individual.

We also vary the number of training subjects from 2 to 7, which gives a small subject-scaling curve.

As a secondary transfer test, we take the model trained on the Natural Scenes Dataset and calibrate it on CNeuroMod-THINGS participants. That dataset has 4 densely scanned people and noise ceilings above 70% in visual areas. The test asks whether the context encoder handles a new scanner and a new image set without retraining.