Where Does the Brain Speak? Locally Gated Patch Decoding for fMRI Reconstruction

Systems that reconstruct seen images from brain scans now produce sharp, photographic pictures. Two failures remain. Nobody can say which parts of a given picture actually came from the brain recording and which parts the image generator made up. The usual scores also disagree with human viewers about which reconstructions are better. This note argues that both failures share one cause. The brain signal constrains some regions of an image far more than others, yet decoders and metrics treat the image as one global object.

We propose to measure and control reconstruction region by region. The image is split into overlapping windows. In each window we compute how much of the reconstruction's fidelity disappears when the true scan is replaced with a mismatched one. That difference is a map of where the brain actually "speaks." We then build a decoder that uses this idea at generation time. Regions that the scan reliably constrains are locked to brain-derived structure. Regions it does not constrain are left to the generator and openly labelled as prior-generated. The bet is that the objective-subjective quality gap sits mostly in the low-reliability regions. If so, it can be closed there without weakening the guarantee that every brain-attributed detail is traceable to the scan.

Core Claim

The gap between metric scores and human preference in brain decoding is spatially concentrated, and a region-resolved notion of provenance (where an image detail came from) can separate it. Two pieces of evidence motivate the claim. MindEye2 decodes CLIP embeddings and renders them with SDXL. It found that CLIP similarity favoured its "unrefined" reconstructions, while human raters strongly preferred the caption-refined ones. Brain-Measurable Diffusion Decoding bans all target-side help at test time, such as target captions, retrieval pools, or picking the best of N samples by closeness to the target. It verifies provenance by showing that identification falls to chance when scans are shuffled. That check is global, however: it shows the image as a whole depends on the scan, but not which parts do.

Our hypothesis is that human preference for "refined" images comes from realistic texture added in regions the scan barely constrains, such as the periphery, backgrounds, and fine texture. Fidelity metrics penalise that added texture because they assume a single ground truth, the "fallacy of the single ground truth" that Yun Cao et al. describe for one-to-many tasks. If the hypothesis is true, a decoder that frees the generator only where the brain is silent should win human preference while keeping brain-attributable fidelity. If the gap is instead spread evenly across the image, the local view adds nothing and we will report that.

Research Program

The first component is a metric. We compute a local fidelity score $F(u; \hat{x})$ at each image location $u$. The score combines a perceptually filtered squared error, following PAMSE (which Gaussian-smooths both images before taking squared error, mimicking human contrast sensitivity), with a low-layer deep-feature distance. Both are evaluated in overlapping windows and blended with distance-decay weights, as in Zhou et al. and Eggimann et al.. Blending matters because hard tiling would create seams in the map at window edges, the boundary artifact catalogued by Yang et al..

The brain-attributable fidelity map is defined as

$$A(u) = F\big(u;\hat{x}(r)\big) - \mathbb{E}_{r'}\Big[F\big(u;\hat{x}(r')\big)\Big],$$

where $\hat{x}(r)$ is the reconstruction from the true scan $r$, and $r'$ ranges over scans recorded for other stimuli. $A(u)$ is the fidelity at $u$ that the specific scan explains, over and above what the generator would produce from any plausible scan. Summing $A$ over the image gives a provenance-corrected global score. Splitting the sum by reliability tier gives the test of the core claim.

The second component is a decoder. Its starting point is the three-channel, brain-measurable SD1.5 decoder of Brain-Measurable Diffusion Decoding. That decoder has a VAE-latent structural anchor, a CLIP-aligned semantic code, and cross-attention tokens trained through the frozen UNet. We change it in three ways.

First, each early-visual-cortex voxel is assigned a receptive-field location by fitting an encoding model on training images, so voxels can be grouped by which image region they "see." The voxel grouping follows the spirit of the shared functional clusters in Brain-IT. The structural anchor and a local token set for each overlapping latent tile are then predicted mainly from voxels whose receptive fields cover that tile. A global semantic token set, decoded from higher visual cortex, is shared by all tiles.

Second, denoising follows UniMIE and HD-Fusion. Every tile's noise estimate is computed separately and the overlapping estimates are averaged at every reverse step, so tiles reach agreement throughout sampling rather than being stitched afterwards. The reason for tiles is not memory, since NSD reconstructions are only 512 pixels wide. The reason is that tiles let each region carry its own brain-derived conditioning through an unmodified UNet.

Third, each tile gets a reliability weight computed only from the scan. The weight is the agreement among an ensemble of structural decoders fit on bootstrap resamples of the training data. Where the ensemble agrees, the tile is initialised from the structural anchor with strong guidance and little added noise. Where it disagrees, the tile starts closer to noise and relies on the global semantic tokens. The output is a reliability map shipped alongside every reconstruction. Because every control is a function of the scan, fixed weights, and target-independent randomness, the decoder still meets the brain-measurable criterion.

A third, smaller strand tests whether training the structural channel with the PAMSE loss instead of plain squared error reduces the blur that Güçlü et al. and Alahi et al. trace to averaging over possible edge positions. PAMSE's reported advantage comes from synthetic distortion datasets. On TID2008 it reaches a Spearman rank correlation of 0.9162 with human scores, against 0.7718 for MSE. It has not been tested on generative outputs, so we treat this strand as an open question rather than a given.

Baselines

The naive reference is the unmodified global, single-pass, three-channel decoder from Brain-Measurable Diffusion Decoding, retrained by us on the same NSD subjects and splits. It defines the "do nothing" level of both fidelity and preference. A shuffled-scan version of every system gives the null used in $A(u)$.

The strongest brain-measurable competitor is that same paper's reported model. On NSD subject 01, over the 982 shared test images, it reports CLIP two-way identification of 99.1% and AlexNet-2 identification of 98.2%. Its absolute PixCorr and SSIM values are not given in the file, so we will rerun it. Beating it means two things: matching its identification within noise and its global brain-attributable fidelity, while achieving a significantly higher human preference rate in blind pairwise comparison.

On the perceptual side, the strongest competitor is MindEye2's refined pipeline. The file gives no numbers for refined-versus-unrefined metric scores, only the qualitative finding. Its caption-based refinement is decoded from the scan, so it can be run fairly. We will run it on identical subjects and report its $A(u)$ maps. Our prediction is that refinement raises preference by adding non-attributable content even in high-reliability tiles, whereas our decoder confines that content to low-reliability tiles.

Brain-IT reports beating baselines on 7 of 8 standard metrics with one hour of data. No exact values are in the file, so it enters as a rerun reference for structural fidelity. MindHier reports 96.4% CLIP and 95.9% Inception scores on NSD and serves as the non-diffusion reference.

Four ablations isolate the key ingredient: uniform reliability (all tiles gated equally), tiles without receptive-field routing (every tile sees all voxels), a single-pass global decoder with a spatially varying guidance mask instead of tiles, and MSE versus PAMSE on the structural channel. All systems share the decoder backbone, training data, a fixed tuning budget of equal hyperparameter trials, and one rendering per scan.

Why not just…?

Why not just use a spatial guidance mask in one global pass instead of tiles? This is the most serious threat. Varying guidance strength per pixel and masking cross-attention by region could reproduce most of the effect without overlapping tiles. Our decoder-side argument for tiles is narrow: they deliver different conditioning tokens to different regions through an unmodified UNet, and they average noise estimates rather than masking attention. The masked single-pass ablation settles it. If it matches the tiled decoder on preference and $A(u)$, we drop tiling from the decoder, keep overlapping windows only in the metric, and say so.

Why not just pick among several samples using a human-preference model? Choosing the best of N samples with HPS uses no target information, so it is brain-measurable. It raised user preference to 74% against SD v1.4 in that paper. The cost is the alignment tax HPS itself documents: it rewards pleasing images regardless of the condition. We run it at matched N and expect it to raise preference while lowering $A(u)$ in high-reliability tiles. If it does not lower $A(u)$, it becomes the preferred simple method.

Why not just rely on the global shuffled-scan check? Global identification near 99.1% shows that an image depends on the scan, not which pixels do. A decoder could pass that check through correct gist alone while inventing every local structure. Only a local map can show this, and the human annotation study below tests whether $A(u)$ tracks what people see.

Why not just use the spread across diffusion samples as the uncertainty map? Spread across samples measures how varied the generator's prior is, not how much the scan constrains each region. A confident hallucination has low spread. We compare sample spread against our bootstrap-ensemble reliability as predictors of held-out local fidelity.

Proposed Experiments

All decoding work uses NSD, training on subject-specific sessions and testing on the shared test set. The full-data regime and the one-hour held-out-subject regime of MindEye2 are both included. Out-of-distribution checks use NSD-Synthetic and THINGS-fMRI. We report the standard eight-metric suite, $A(u)$ overall and by reliability tier, and blind human pairwise preference.

The first decisive check is a local human study. Raters view a stimulus next to a reconstruction and mark the regions that match or contradict it. We then ask whether $A(u)$ predicts those marks better than local PixCorr, local CLIP similarity, or HPS heatmaps. If it does not, the metric fails, independent of the decoder.

The second check tests the core claim directly. Across all systems, we regress human preference differences on the change in $A$ within high- and low-reliability tiles. The claim predicts that preference gains load on the low-reliability tiles. If gains load on high-reliability tiles, the gap is not spatially separable and the gating idea is wrong.

The third check is the decoder head-to-head against the baselines and ablations above. Success requires identification and global $A$ within noise of Brain-Measurable Diffusion Decoding, and a significant preference gain over it. The comparison against the masked single-pass ablation decides whether overlapping tiles belong in the decoder at all.