Reading the gist of continuous speech from a person's brain scan is limited mostly by data, not by models. A useful decoder needs many hours of scanning for each person, but a new participant can give perhaps an hour. Text, by contrast, is effectively unlimited. Encoding models are models that predict the brain's response to a stimulus. Once such a model is fit, it can turn any amount of text into simulated brain responses for that person, and those synthetic scans can train the decoder. This is the same trick as back-translation in machine translation, where a reverse model manufactures training pairs from unpaired monolingual text.
The obvious version of this fails for a reason the unpaired image-translation literature already identified. An encoding model's prediction is a clean average response. Real fMRI is dominated by noise the stimulus does not explain. A decoder trained on clean synthetic scans learns cues that are absent from real ones. The same mismatch in information content is what lets cycle-consistent translators hide information in their outputs instead of learning true correspondences. We propose to make synthetic fMRI exactly as informative as real fMRI, no more, by adding noise sampled from the subject's measured residual structure. We then test whether this noise-matched back-translation lets a decoder for a new subject, trained on one hour of scans, approach one trained on ten.
Core Claim
Our main claim has three parts. A sequence decoder from fMRI to coarse linguistic tokens can be trained largely on synthetic fMRI generated from unpaired text. The benefit depends on matching the synthetic data's signal-to-noise structure to the real data. Without this matching, back-translation will help little or even hurt, because the decoder overfits to noiseless predictions.
The decoder is trained with connectionist temporal classification (CTC). CTC is a loss that trains a model to output a token sequence from a longer input stream without being told which input frame corresponds to which token. It does this by summing over every possible monotonic alignment, using a special "blank" symbol for frames that emit nothing. CTC suits fMRI because the delay between a word and its response varies across brain regions and people (roughly 5 to 10 seconds). With CTC, we never have to commit to a fixed lag.
The synthetic data come from a voxel-wise encoding model, meaning a regression from deep-network features of the stimulus to each voxel's measured response. Let $\hat f_s(x)$ be subject $s$'s encoding-model prediction for stimulus $x$. We generate
$$\tilde b = \hat f_s(x) + \eta, \qquad \eta \sim \mathcal{N}(0, \Sigma_s),$$
where $\tilde b$ is a synthetic voxel time series and $\Sigma_s$ is the covariance of the encoding model's residuals on held-out real runs. We estimate $\Sigma_s$ as low rank plus diagonal across voxels, with temporal autocorrelation. The new ingredient is $\Sigma_s$, not the encoder. It makes the synthetic data carry about as much stimulus information as the subject's real data, which is bounded by the noise ceiling.
Relation to Prior Work
The closest supplied work on decoding is Cross-brain transfer of high-performance intracortical speech and handwriting BCIs. That paper trains a recurrent CTC decoder on implanted motor-cortex electrodes and shows that per-user linear input layers enable transfer from other users when the target has fewer than 200 sentences. We reuse its method (CTC plus per-subject input layers) as our strongest transfer baseline. Our setting differs in three ways. The signal is slow and hemodynamically blurred rather than spike-rate. Transfer comes from other people's brains. Our extra data comes from text no one heard in a scanner.
Two lines of fMRI work motivate the data-efficiency gap but attack it through cross-subject sharing on static images. The Wisdom of a Crowd of Brains learns voxel embeddings that query image features and adapts to a new subject from about 100 images. MindAligner maps a new subject's voxels onto a reference subject with a low-rank transfer matrix, using about one hour of data. Neither handles continuous speech, and neither manufactures training data. Graph-Based Decoding Model aligns subjects without synchronized stimuli but works on discrete cognitive states.
The encoder side is supplied by TRIBE. TRIBE predicts whole-brain responses to movies from frozen text, audio, and video features, and its attention learns the hemodynamic lag. It also supplies the benchmark we use: The Algonauts Project 2025 Challenge. Phonemic segmentation of narrative speech supplies the narrative-speech dataset and the banded ridge methodology for a lighter, per-subject encoder.
The constraint on our design comes from unpaired translation. Learning to Translate Images Without Paired Data frames cycle consistency, a loss requiring that translating there and back recovers the input. The research file attributes to this line of work the warning that cycle consistency breaks down when the two domains carry unequal information, because the generator can hide recoverable signals in its output. Text-to-fMRI-to-text is exactly such an unequal cycle. This warning shapes two of our choices: we freeze the encoder, and we explicitly test the joint-training version that cycle methods would suggest.
Research Program
The pipeline has three parts.
First, for each target subject we fit an encoding model on that subject's limited real data. The model consists of frozen language-model and audio features, a learned lag, and per-voxel ridge weights. In the CNeuroMod setting, the encoder can also include shared layers pretrained on the other subjects, in the style of TRIBE's subject-conditional heads. Residuals on held-out runs give $\Sigma_s$.
Second, we build the decoder. It applies a per-subject linear input layer to voxel time series, followed by a shared temporal Transformer. Its output is upsampled by a factor $r$ and trained with CTC against a target sequence of semantic tokens. These tokens come from clustering contextual language-model embeddings of content words into a vocabulary of a few hundred units.
The upsampling and the coarse tokens address a known hard limit of CTC: the input must have at least as many frames as the output has tokens. With a repetition time of one to two seconds, fMRI has far fewer frames than a speech stream has phonemes or words. We choose $r$ so that $rT \ge U$ holds in every training window, where $T$ is the number of fMRI volumes and $U$ the number of target tokens. We report the fraction of windows where it would otherwise fail.
Third, we generate synthetic data. We pass large text and audio collections that the subject never heard through the encoder, add noise drawn from $\Sigma_s$, and train the decoder on a mix of real and synthetic windows. The mixing ratio is tuned on validation runs.
The scientific analysis asks what the noise model must capture. We compare four noise models: no noise, white noise scaled to each voxel's residual variance, spatially correlated noise, and spatially plus temporally correlated noise. We relate decoding gains to how closely a held-out discriminator can tell synthetic from real windows. The gap between real and synthetic data should predict when back-translation helps.
We also run the cycle experiment the unpaired-translation literature invites. We unfreeze the encoder and train it jointly with the decoder on the round-trip loss text → synthetic fMRI → text. We then measure whether its prediction accuracy on real held-out fMRI falls while round-trip accuracy rises. That pattern would be direct evidence of the encoder smuggling information into the synthetic scans.
Baselines
The naive reference point is a decoder that ignores the brain. It emits the most likely token sequence under a language prior fit to the training transcripts, with sequence length set from the window duration. Any brain decoder must beat it on token error rate and on identification accuracy, or it has learned nothing from fMRI.
The simplest brain baseline is a lagged ridge decoder. It shifts the voxel time series by a fixed hemodynamic delay, regresses each volume onto the average token embedding in the matching window, and decodes by nearest token. This is the decoding analogue of the standard encoding pipeline in Phonemic segmentation of narrative speech. It tests whether CTC's alignment-free training is worth anything at all.
The strongest existing competitor is cross-subject transfer as in Cross-brain transfer of intracortical BCIs. The CTC decoder is pretrained on the other subjects' full real data, then fine-tuned on the target's limited data through a new per-subject linear input layer. That paper found such input layers essential and transfer otherwise worse than training from scratch.
A second strong competitor is encoder-inversion decoding. A language model proposes candidate token sequences, and the target subject's encoding model scores them by the likelihood of the observed fMRI. This uses the same encoder we use for synthesis but trains no decoder.
All methods receive the same target-subject real data, the same frozen features, the same encoder where one is used, and the same hyperparameter search budget, measured in validation-set trials and GPU hours. The two baselines that use no synthetic data are allowed equal total training steps.
There are two key ablations. The first is back-translation with noiseless synthetic data, which isolates $\Sigma_s$. The second is back-translation without the cross-subject pretraining, which tests whether synthetic data and transfer are complementary.
The Algonauts file gives encoding numbers, which we use to check our encoder. The challenge's linear baseline reaches $r = 0.20$ in distribution and $r = 0.09$ out of distribution (Algonauts 2025). TRIBE reaches a mean Pearson correlation of 0.2146 (TRIBE). For continuous text decoding from fMRI on CNeuroMod or the Moth dataset, no reported result was found on Lacuna. The decoding baselines above must therefore be run by us.
Two criteria count as success. First, at one hour of target data, the noise-matched method must close at least half the gap between transfer-only and transfer-plus-ten-hours on token error rate, with non-overlapping subject-level confidence intervals. Second, the noiseless ablation must not do so.
Why not just…?
Why not just use cross-subject linear input layers, as the BCI work does? This is a real threat. Linear alignment layers were sufficient in Cross-brain transfer of intracortical BCIs. MindAligner gains substantially with a low-rank linear map from one hour of data. We do not replace transfer. We stack synthetic data on top of it. If transfer alone matches transfer plus noise-matched back-translation across the data-size sweep, the proposal's claim is false, and the experiment is built to show that.
Why not just invert the encoding model at test time instead of training a decoder? Encoder inversion uses exactly the information our synthetic data carry. With a strong language-model proposer, it could match an amortized decoder. Our argument is that the inversion search must cover the space of possible sentences for every test window. The decoder, by contrast, absorbs language-model statistics and many hours of synthetic examples at training time. This is an empirical question, settled by running both with the same encoder and language model. We also report a hybrid that rescores the decoder's beam with the encoder, in case the two are complementary.
Why not just align subjects with CycleGAN-style unpaired translation between their brains? In encoding and decoding the stimulus is always known. Frozen foundation-model features therefore already provide the shared coordinate system that cross-brain translation would try to learn. The supplied evidence on cycle methods also predicts that round trips between noisy, unequal brain domains will encode non-repeatable noise. We use unpaired translation as a diagnostic, the joint-cycle experiment, rather than as the method.
Why not just shift by a fixed lag and skip CTC? If the lagged ridge or a lagged framewise classifier matches CTC, alignment-free training buys nothing in fMRI, and we would report the back-translation result with the simpler decoder. CTC is justified only if lag varies enough across regions and subjects that a fixed shift loses information. Comparing the two decoders on identical data tests this directly.
Proposed Experiments
The primary setting is CNeuroMod via Algonauts 2025, with four subjects. For each subject in turn, the other three provide full training data. The target receives 1, 2, 5, or 10 hours of Friends seasons 1–6. We test on the in-distribution season 7 set and the out-of-distribution novel movies. Unpaired text comes from dialogue transcripts and spoken-language text that never appeared in the scanner, rendered to audio features with a speech synthesizer for the audio stream.
The secondary setting is the Moth narrative-speech dataset (11 subjects, about two hours each). This tests speech without video at the low-data extreme, with a per-subject banded ridge encoder.
The metrics are three. The first is semantic token error rate. The second is identification accuracy: picking the true 30-second window from 1,000 candidates by similarity between decoded and true token embeddings, following the identification protocol in The Wisdom of a Crowd of Brains. The third is encoder Pearson correlation, checked against the reported Algonauts figures.
The decisive checks are three. First, the noise-matched minus noiseless difference in decoding gain should be positive and grow as target data shrinks. Second, decoding gain should track how indistinguishable synthetic windows are from real ones across the four noise models. Third, the joint-cycle variant should show encoder accuracy on real data dropping while round-trip accuracy rises. Each outcome has a clear negative reading: if any of these fails, we learn that synthetic fMRI is either redundant with transfer or untrustworthy as training data.
References
- Cross-brain transfer of high-performance intracortical speech and handwriting BCIs
- The Wisdom of a Crowd of Brains: A Universal Brain Encoder
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
- Graph-Based Decoding Model for Functional Alignment of Unaligned fMRI Data
- TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction
- The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
- Phonemic segmentation of narrative speech in human cerebral cortex
- Generative Adversarial Networks: Recent Developments