Brain decoders turn recorded brain activity into images or text by mapping it into the embedding space of a frozen generator such as Stable Diffusion. They work well for the people they were trained on and fail for anyone new. Each person's cortex is folded and tuned differently, so the same picture produces voltage or blood-flow patterns that cannot be compared coordinate by coordinate across people. Today a new person needs either hours of their own scans or a fitted linear map into another subject's space.
We propose to skip fitting entirely. Show the new person a small, fixed set of "anchor" images that every training subject also saw. Then describe each of their brain responses only by how similar it is to their own responses to those anchors. That similarity profile does not depend on how a particular brain lays out its signals, so one decoder trained on similarity profiles from many people can read out a new person's profile directly. The new person contributes a few dozen to a few hundred anchor trials and no gradient steps.
Core Claim
The idea comes from machine learning work on latent space misalignment, meaning the fact that two independently trained networks encode the same inputs in incompatible coordinate systems. Relative representations enable zero-shot latent space communication showed that re-expressing each vector as its cosine similarities to a set of anchor samples cancels arbitrary rotations and rescalings. This made it possible to plug an encoder from one network into a decoder from another with no training. Their test cases were artificial networks sharing inputs. Our "networks" are human brains, and the shared inputs are stimuli.
For subject $s$ with preprocessed response $x$ (an fMRI voxel pattern or an EEG channel-by-time window), we define the relative code
$$r_s(x) = \big[\,\mathrm{corr}(W_s x,\; W_s \bar a_{s,j})\,\big]_{j=1}^{N}$$
Here $\bar a_{s,j}$ is subject $s$'s response to anchor image $j$, averaged over repeated presentations, and $N$ is the number of anchors. $W_s$ is a whitening transform estimated only from that subject's anchor responses, using a shrinkage covariance estimate. A single decoder $g$, trained on pooled source subjects, maps $r_s(x)$ to the CLIP image embedding that conditions a frozen diffusion generator.
The claim we test is a sharp one. Once $W_s$ and the anchor responses are fixed, $r_s$ is a linear map defined by data with no fitted parameters. The competing approach is a linear adapter fit on the same $N$ trials. We predict the anchor code beats the fitted adapter when $N$ is small, and that the fitted adapter catches up as $N$ grows. The value of the proposal lies in where that crossover falls. If it falls near $N \approx 1000$, the scale at which Through their eyes reported ridge regression matching a full 20-hour dataset, the approach is useless and we will say so. If it falls well below that, calibration shrinks from hours to minutes.
Why this is not already done
Several nearby lines of work stop short of this. Relative geodesic representations replace cosine similarity with distances along curved manifolds and reach ImageNet retrieval MRR 0.566 versus 0.221 for cosine. However, their distance needs a "pullback metric" from a trained output head, and raw brain recordings have no such head. We therefore keep correlation on the brain side and treat geodesic distances only as a possible refinement on the CLIP side.
Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching recovers rotations between spaces without any pairs. It assumes the two spaces have equal dimension and differ only by a rotation, and neither holds across brains. Through their eyes fits ridge and Procrustes maps between subjects' voxel spaces and finds that Procrustes overfits. That is the fitted-map family we compete against, not a version of our method. ViEEG (ViEEG) trains a subject-independent EEG encoder with no calibration at all. Human-Aligned Image Models changes the target embedding space rather than the brain side.
The motivating evidence is Alignment between Brains and AI. Its large set of CKA measurements on NSD shows that similarity structure converges across brains and models. That is exactly the invariant a relative code exploits, but the paper builds no decoder. Our proposal turns that analytical observation into a calibration procedure.
Research Program
The system has three parts, each small by design, following the evidence from ENIGMA and Scalable VLM training that light projectors into frozen backbones are the stable regime.
The first part is the anchor protocol. We fix an anchor pool of stimuli seen by every subject and order it so that any prefix of length $N$ covers diverse concepts, using greedy farthest-point selection in CLIP space. Diversity matters because Relative representations report collapse with too few or poorly spread anchors.
The second part is the pooled decoder $g$. It is a small multilayer network from $r_s(x)$ to CLIP embeddings, trained on source subjects with the standard InfoNCE contrastive loss plus a regression term. We randomize $N$ during training so that a single model serves every anchor budget.
The third part is the frozen output side. On NSD, predicted CLIP embeddings condition the Brain-Diffuser pipeline used by Through their eyes. On THINGS-EEG, they are scored by 200-way zero-shot retrieval. As an ablation of the output target, we also train $g$ to predict the CLIP-side relative code. That code is then mapped back to an absolute CLIP embedding by an inverter trained on images alone, with no brain data. This tests whether expressing both sides in the same anchor coordinates helps.
The analysis the program exists to produce is decoding accuracy on a held-out new subject as a function of $N$, for our method and for each fitted alternative on identical trials. For EEG, whose single trials are noisy, we additionally measure how accuracy depends on the number of repetitions averaged into each $\bar a_{s,j}$. Noisy anchors are the most likely way this method fails.
Baselines
The naive reference point is the zero-calibration decoder: train on source subjects and apply to the new subject untouched. On THINGS-EEG the strongest reported number for this setting is ViEEG's leave-one-subject-out 200-way Top-1 accuracy of 22.9% (ViEEG). We re-run ViEEG's subject-independent setup under our splits as the floor. Any anchor budget that fails to clear it means the calibration trials are wasted.
The upper reference is full subject-specific training. ViEEG reports 40.9% Top-1 on THINGS-EEG, against 27.3% for NICE. The gap between 22.9% and 40.9% is the headroom our method must recover with $N$ anchors.
The strongest fitted competitor on fMRI is ridge regression from the new subject's voxels into a source subject's voxel space, followed by the frozen Brain-Diffuser, as in Through their eyes. With 982 shared images this matched the full-data decoder in reconstruction quality. The file reports no reconstruction metric value for this baseline on Lacuna, only the 90% data reduction. We therefore re-run it ourselves at every $N$ on the same anchor trials, with ridge strength chosen by nested cross-validation inside the anchors.
The second fitted competitor is a subject-specific linear adapter from the new subject's responses into the input of a pooled decoder, trained on the same $N$ trials with the same tuning budget as our decoder's hyperparameter search. On EEG we build it on top of ViEEG's encoder.
The key-ingredient ablations are three: our method without whitening, with single-trial rather than averaged anchors, and with random rather than diversity-ordered anchors.
Beating the baselines means strictly higher held-out Top-1 retrieval on THINGS-EEG and higher CLIP two-way identification on NSD than both fitted maps at the same $N$, for at least $N \le 200$, with confidence intervals from resampling subjects.
Why not just…?
Why not just fit ridge regression on the anchor trials? Ridge regression is the real threat, and Through their eyes shows it works at about 1000 trials. Mapping roughly fifteen thousand voxels onto another subject from 50 to 200 trials is badly underdetermined, and regularization shrinks it toward an uninformative map. The relative code instead uses the anchors as the map itself. The $N$-sweep settles which effect wins. If ridge ties at small $N$, the proposal reduces to a cheaper reimplementation and we report that.
Why not just fine-tune a small adapter into a pooled decoder? This is the standard practice, and it is identical to ours at large $N$. Our method is the special case in which the adapter weights are the whitened anchor responses rather than learned values. The honest prediction is a crossover, and the experiment is designed to locate it rather than to claim uniform superiority.
Why not just train a better subject-independent decoder and use no calibration? ViEEG's 22.9% versus 40.9% shows that a large gap remains without any subject data. Seeing the imagined shows that a latent alignment step beats plain voxel-space ridge for out-of-distribution signals. Calibration-free decoding is our floor, not a substitute.
Why not just align subjects without anchors using hyperspherical matching? HGA needs spaces of equal dimension that differ only by a rotation. We will test it where it is applicable, between the outputs of subject-specific encoders of equal dimension. We expect failure because brains differ by more than a rotation, but this is an empirical check, not an assumption.
Proposed Experiments
On NSD we split the shared images into a disjoint anchor pool and a held-out evaluation set. We rotate each of the four subjects in as the new subject, sweep $N$ from 25 to about 900, and report CLIP-based two-way identification and retrieval of Brain-Diffuser reconstructions. On THINGS-EEG we use the same leave-one-subject-out rotation, with anchors drawn from training-set images, and report 200-way zero-shot Top-1 and Top-5 retrieval. As a secondary target we swap CLIP for a human-aligned embedding following Human-Aligned Image Models. The relative code should transfer better when the target space shares the brain's similarity structure.
Three checks are decisive. First, where the accuracy-versus-$N$ curve of our method crosses that of ridge and of the fitted adapter. Second, whether EEG accuracy saturates with anchor repetitions before our method overtakes the floor of 22.9%. Third, a representational check: across subject pairs, the correlation between relative codes for the same held-out stimulus should exceed that for different stimuli. If it does not, the invariance we rely on is absent and the decoder results should not be trusted.
References
- Relative representations enable zero-shot latent space communication
- Through their eyes: Multi-subject brain decoding with simple alignment techniques
- Connecting Neural Models Latent Geometries with Relative Geodesic Representations
- Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching
- ViEEG: Hierarchical Visual Neural Representation for EEG Brain Decoding
- Human-Aligned Image Models Improve Visual Decoding from the Brain
- Alignment between Brains and AI: Evidence for Convergent Evolution across Modalities, Scales and Training Trajectories
- ENIGMA: EEG-to-Image in 15 Minutes Using Less Than 1% of the Parameters
- Scalable Vision Language Model Training via High Quality Data Curation
- Seeing the imagined: a latent functional alignment in visual imagery decoding from fMRI data