Models that predict brain activity from images ("encoding models") are usually fit one voxel at a time. Each small cube of cortex measured by fMRI gets its own linear regression onto thousands of deep-network features. That design needs tens of hours of scanning per person, and it throws away something we already know about visual cortex: nearby tissue has similar selectivity, and the layout of that selectivity (orientation maps, face and scene patches, animacy gradients) is broadly shared across people. We propose to fit a new person's visual cortex differently. Instead of learning a weight vector for every voxel, we take a network whose units already sit on a simulated cortical sheet and learn only a smooth geometric warp that lays that sheet onto the person's cortical surface. Each voxel then reads out from the model units that land near it.
The claim is that this turns cross-subject encoding from a problem with millions of parameters into one with a few hundred. With an hour or less of data, it should predict a held-out person's responses better than per-voxel regression. Because the readout has so few parameters, it should also fail less badly when test images differ from training images. A second payoff is scientific. The fitted warps give a compact, comparable description of how one person's functional map differs from another's. Much of the literature on cortical organization rests on three to eight deeply scanned individuals, so the warps offer a direct way to test which organizational claims are universal.
Core Claim and What Is New
The mechanism comes from A Unifying Principle for the Functional Organization of Visual Cortex. That paper introduced the topographic deep artificial neural network (TDANN): a ResNet-18 whose units are assigned positions on a 2D sheet sized to human V1 and ventral temporal cortex (VTC), and which is trained with a self-supervised objective plus a loss that penalizes distant units for responding alike. TDANN reproduces V1 pinwheel density and produces face, body and scene patches that match Natural Scenes Dataset (NSD) data in count, size and relative geometry. Its units also correlate one-to-one with human voxels better than those of non-topographic networks. TDANN was evaluated only on in-distribution test splits, with no restriction on neural data and no transfer across subjects. We reuse it as a prior for mapping, and we do not propose a new topographic network.
Concretely, let $x_v$ be the position of voxel $v$ on a flattened patch of a subject's cortical surface, $p_u$ the sheet position of model unit $u$, and $f_u(s)$ that unit's response to image $s$. We predict
$$\hat r_v(s) = a_v \sum_u \exp\!\left(-\frac{\lVert \phi_i(x_v) - p_u \rVert^2}{2\sigma^2}\right) f_u(s) + b_v,$$
where $\phi_i$ is a smooth warp for subject $i$ and $\sigma$ is a readout width shared within an area. The warp is a thin-plate spline with a few dozen control points per area. The only per-voxel quantities are a gain $a_v$ and an offset $b_v$. Fitting a new subject means estimating the warp control points, $\sigma$, and two scalars per voxel. Ridge regression on the same network would instead estimate thousands of weights per voxel.
The idea sits between two lines of cross-subject work. MindTuner adapts a multi-subject decoder to a new person with a small low-rank module and one hour of NSD data. Its adaptation weights concentrate in higher visual cortex, and it provides no encoding model of how cortex is laid out. Brain-tuning jointly fine-tunes speech models on several people's fMRI projected to a common surface, reaching the same alignment with 20% of the data and transferring to held-out listeners. It has not been tried on visual cortex. Both methods adapt the features and treat cortical position as fixed anatomy. We freeze the features and adapt where on the cortex each feature lives. That is the axis along which people's visual maps are known to differ.
Research Program
The first stage builds area-matched sheets. TDANN defines sheets for a V1-like layer and a VTC-like layer. We warp the early layer onto NSD's V1 and V2 and the late layer onto VTC, using the retinotopic and category ROIs defined by NSD. Intermediate areas such as V4 have no dedicated sheet. We will test the intermediate layers there, and we report V4 as exploratory, because curvature domains of the kind mapped in human V4 with layer fMRI and macaque V4 with calcium imaging (A large calcium-imaging dataset reveals a systematic V4 organization for natural scenes) are finer than NSD's resolution.
The second stage fits the warp. For each area we initialize $\phi_i$ from a group template learned on the training subjects. We then optimize the control points by gradient descent on prediction error, with a bending-energy penalty on the warp. Because the Gaussian readout is differentiable in $\phi_i$, the fit is a small nonconvex problem. We will use multiple restarts from rotated and reflected initializations, since maps can plausibly be mirrored relative to anatomy.
The third stage uses the warps for science. We will ask two questions. First, do warps fitted from a small fraction of a subject's data locate that subject's face, body and scene patches, as defined from their full data? Second, does the between-subject variance in warp parameters shrink or grow from V1 to VTC? We then apply the same machinery to THINGS-fMRI. There, Principles of coarse-scale functional organization in occipitotemporal cortex found smooth maps for all 15 object properties it tested, with no single dominant dimension, in three participants. We will test whether a warp learned on NSD subjects predicts where those property gradients fall in THINGS subjects. If it does, the multi-property organization is shared structure rather than three idiosyncratic brains.
Baselines
The naive reference is anatomical alignment alone. We fit per-voxel ridge regression on each training subject, resample the weights to a common surface template, average them, and apply the average to the held-out subject with no fitting at all. This zero-data point tells us how much person-specific geometry is worth adding. It is also a serious competitor in V1 and V2, where anatomy predicts retinotopy well.
The strongest standard competitor is per-voxel ridge regression on the new subject's calibration data. We run it on the same frozen TDANN features, and separately on the best-performing non-topographic network we can find on the full data, so that feature quality is not confounded with readout design. The protocol follows the Algonauts 2023 benchmark, whose AlexNet baseline reached a mean noise-normalized prediction accuracy of 40.48% across subjects with full per-subject data. We add a ridge variant initialized from and shrunk toward the anatomically aligned group weights, which is the fair version of "use the other subjects."
The second strong competitor transfers multi-brain tuning to vision. It jointly fine-tunes the feature network with low-rank adapters and a shared head across the seven training subjects on a common surface, as in Brain-tuning, and then adapts to the held-out subject with the same calibration data. We found no reported result on Lacuna for cross-subject visual encoding accuracy at one hour of data. MindTuner's 83.1% image retrieval at one hour is a decoding figure and is not comparable. Every method gets the same images, the same calibration budgets and the same hyperparameter search budget on a validation split of the calibration data.
The key ablation isolates the topographic prior. We run the identical warp readout on a non-topographic ResNet-18 trained with the same self-supervised objective. Its units get sheet positions assigned after training, by placing them to maximize response smoothness. If the gains come only from having a low-dimensional spatial readout, this ablation will match the full method. If topography learned during training matters, it will fall short. Beating a baseline means higher noise-normalized accuracy on held-out images, averaged over leave-one-subject-out folds, with a paired test across the eight subjects at each calibration budget.
Why not just…?
Why not just smooth the ridge weights over the cortical surface? A spatial smoothness prior on per-voxel weights, with no model sheet, is the most likely rival. It encodes the same fact that neighbors are similar. It does not encode which feature belongs where, so with little data it still has to discover the map. The ablation above, together with a surface-smoothed ridge baseline run under the same budgets, settles the question. If surface-smoothed ridge matches the warp readout below one hour of data, the TDANN prior adds nothing, and we would report the method as a regularization result.
Why not just fine-tune the features across subjects, as brain-tuning and MindTuner do? Feature adaptation may well win once calibration data exceed a few hours, and we expect a crossover. Adapter-based methods still add millions of parameters, roughly 12.3M in MindTuner, whereas the warp adds hundreds. The low-data end of the calibration curve therefore shows whether geometry or features is the bottleneck. We also test a combination, warp plus adapters. If feature tuning dominates at every budget, the warp contributes interpretability but not accuracy, and we would say so.
Why not just rely on anatomical registration? In early visual cortex this is a real threat, and we expect little gain there. The case for learned warps rests on VTC. There, Silson et al. show that ventral cortex carries weak retinotopic structure. Category patches are also what MindTuner found most idiosyncratic across people. The prediction is specific: the gain over anatomical transfer should be near zero in V1 and largest in VTC. If the gain is uniform or absent, the hypothesis is wrong.
Proposed Experiments
The main experiment is leave-one-subject-out on NSD. For each held-out subject we give every method calibration subsets from a few hundred images up to the full set, then plot noise-normalized accuracy on the shared test images against calibration size, separately for V1, V2, V4 and VTC. The decisive check is whether the warp readout's curve lies above both ridge baselines in VTC at budgets of one hour or less, and whether the TDANN version beats the post-hoc-positioned ablation.
The second experiment tests out-of-distribution transfer. It borrows the attribute-based splits of the MacaqueITBench study, which found encoding accuracy collapsing across all eight networks tested, with ViT falling to about 20% of in-distribution accuracy on saturation shifts. We construct low-to-high contrast and saturation splits within NSD images and compare the in-distribution-to-OOD ratio of the warp readout against ridge on identical features. That study could not tell whether the collapse came from the features or from an overfit mapping. Holding features fixed while shrinking the mapping from thousands of parameters per voxel to two answers that question. A much smaller drop for the warp readout implicates the mapping. An equal drop implicates the features.
The third experiment tests transfer across datasets. We fit the group template on NSD, fit warps to THINGS-fMRI participants with a small calibration subset, and correlate the predicted smoothness maps for animacy, size and moveability with those measured from each participant's full data. These are within-subject correlations, and three participants cannot establish universality. The result tests whether a shared warpable map explains organization that was previously described person by person.
References
- A Unifying Principle for the Functional Organization of Visual Cortex
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction
- Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
- Mapping curvature domains in human V4 using CBV-sensitive layer-fMRI at 3T
- A large calcium-imaging dataset reveals a systematic V4 organization for natural scenes
- Principles of coarse-scale functional organization in occipitotemporal cortex
- The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes
- Direct comparison of contralateral bias and face/scene selectivity in human occipitotemporal cortex
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex