Is Higher Visual Cortex Really More Nonlinear, or Just Downstream of More Veins?

Functional MRI does not measure neurons. It measures blood, which drains from capillaries next to active tissue up through small venules into large veins on the cortical surface. A voxel near such a vein therefore records an amplified, blurred mixture of activity from many upstream patches of cortex. Brain-encoding models, the networks that predict each voxel's fMRI response from a deep network's features of the viewed image, ignore this. They fit every voxel as an isolated target, discarding the three-dimensional drainage structure that links voxels together.

That omission matters for a specific, recent scientific claim. Jacobian-Based Interpretation of Nonlinear Neural Encoding Model measures how nonlinear each voxel's response is. Its measure, JNE, is the spread across stimuli of the encoding model's local slopes. On the Natural Scenes Dataset (NSD) it finds nonlinearity rising from primary visual cortex (V1) to face-, body- and place-selective regions. The authors themselves concede that JNE cannot tell neural nonlinearity from vascular nonlinearity. Separately, Vu et al. found large draining veins dominating 38% of right place-area (PPA) voxels and 16% of right face-area (FFA) voxels, exactly the regions where JNE peaks. We propose to build an encoding model whose output side contains an explicit, anatomically directed drainage graph, and to use it to test whether the nonlinearity hierarchy survives once the vascular mixing is modeled.

Core Claim

Veins can manufacture apparent nonlinearity in three separable ways, and each predicts something different.

The first is gain. A vein multiplies the response. Because JNE is a standard deviation of slopes, multiplying a voxel's response by a factor $k$ multiplies its JNE by $k$, so heavily veined voxels score as "more nonlinear" for trivial reasons.

The second is pooling. A downstream voxel sums responses from several upstream patches with different tuning, so its local slope changes depending on which patch the image happens to drive.

The third is saturation. The venous signal compresses at high amplitude, a monotone output nonlinearity unrelated to computation.

Our claim is that a model with all three as separate, constrained components can attribute JNE to neural versus vascular sources. Gain alone is fixed by a normalization. Pooling and saturation require modeling where blood flows from and to.

Concretely, the model predicts voxel $v$'s response to image features $x$ as

$$\hat y_v(x) = g_v\Big(\sum_{u \in \mathcal{U}(v)} A_{vu}\, s_u(x)\Big).$$

Each term is defined as follows. The function $s_u(x)$ is the "neural" response field of source location $u$, produced by a shared readout from deep-network features. The set $\mathcal{U}(v)$ contains $v$ and the locations that drain into it, read off a directed graph built from anatomy. The weights $A_{vu} \ge 0$ are learned drainage weights, with the self-weight free and the upstream weights sparsity-penalized. The function $g_v$ is a learned monotone saturating function.

Neural nonlinearity is then measured as JNE computed on $s_u$, not on $\hat y_v$.

How this differs from prior work

The closest prior corrections are all static or local. Kurzawski et al. regress out a per-voxel scalar gain from curvature, thickness, depth and vein proximity. The contribution of the vascular architecture and cerebrovascular reactivity to the BOLD signal formation across cortical depth normalizes along one-dimensional depth lines using a separate blood-volume scan. Vu et al. use a fixed seven-voxel isotropic neighborhood for phase regression.

None of these lets one voxel's signal be explained by another voxel's tuning along a directed path. None is placed inside an encoding model whose interpretability metric is then re-examined.

Research Program

The work has three stages.

Stage 1: build the drainage graph from data NSD already supports. Nodes are cortical surface locations at the native fMRI resolution. Directed edges run from deeper to more superficial locations, where depth resolution allows. They also run tangentially toward the nearest large-vein location, using the vein-proximity estimate procedure of Kurzawski et al., which they applied to NSD. Edge candidates are pruned to a fixed small in-degree per node and a maximum path length.

The physical motivation comes from three sources. The deep-to-surface drainage hierarchy is described in the cortical-depth paper. The volumetric simulations of Chen & Calhoun show that larger vessels degrade the correspondence between the fMRI image and the underlying neural source. The reverse-labeling DiSpect method of Karasan et al. shows that real drainage basins are three-dimensional and run tangentially, not only straight up.

Building graphs over explicit vessel topology rather than voxel grids follows Graflow, which represents vessel trees as graphs and solves flow by message passing. We borrow the representation, not its rigid-wall flow physics. We use a graph rather than learned dense neighbors because dense k-nearest-neighbor voxel graphs like ViG3D-UNet's carry no notion of flow direction. They are also memory-heavy at fine scales.

Stage 2: establish ground truth in simulation before touching real data. We extend the Chen & Calhoun volumetric phantom in one way. Its random cylindrical vessel forest is replaced by directed synthetic trees of the kind Graflow uses, so that capillary beds actually drain into collector veins.

Neural sources are given known tuning, either linear or nonlinear, in deep-network feature space, driven by real NSD stimuli. We then synthesize voxel responses with controlled gain, pooling and saturation. This answers two questions. First, how much spurious JNE does each vascular mechanism create? Second, does the drainage-aware model recover the planted neural JNE, and does it do so when the anatomical graph is deliberately mis-specified?

Stage 3: apply the model to NSD. We fit it on all eight subjects with a shared feature backbone and recompute JNE and its stimulus-specific variant, JNE-SS, per region on the neural components.

There is one identifiability risk. The model could explain everything as pooling. Three guards address this. Upstream weights are penalized. Self-weights are initialized at one. Every result is reported as a function of the penalty strength.

Baselines

The naive reference is the voxel-independent encoding model with JNE exactly as in Gao et al. It is the published claim being tested. The file contains no per-region JNE values, so we will reproduce them ourselves on NSD with their forward-mode Jacobian code path and the same backbone features. No reported result was found on Lacuna for numerical JNE per region.

The strongest existing correction is Kurzawski-style scalar regression on curvature, thickness, depth and vein proximity, applied to responses before fitting. That paper reports that these factors explain 26% of V1 response-magnitude variance. It also reports that the correction cut the horizontal-to-vertical meridian asymmetry from 49% to 25% on its retinotopy dataset. It is the right competitor because it uses the same anatomical inputs as our graph, minus direction and cross-voxel structure.

A gain-normalized JNE is also run: the standard deviation of the slopes divided by their mean magnitude, which cancels pure multiplicative gain. This is the cheapest possible fix.

Two ablations isolate the key ingredient. In the first, the directed anatomical graph is replaced by an undirected graph with matched degree. In the second, it is replaced by a randomly rewired graph. A third, "saturation only" model keeps $g_v$ but sets all upstream weights to zero.

All variants share the feature backbone, data splits, optimizer, and an identical hyperparameter search budget. Our extra components get no additional tuning trials.

Prediction accuracy is reported on the Algonauts 2023 split as a sanity check, against the challenge's AlexNet baseline of 40.48% mean noise-normalized accuracy (Algonauts 2023). It is also compared with the 61.56% ensemble score of UARK-UAlbany, with single-backbone variants at 54.21–57.01%. We require only that held-out accuracy not fall below the matched single-backbone voxel-independent model.

Our model "beats" the baselines if two conditions hold. In simulation it recovers planted neural JNE where the scalar regression and gain normalization fail. On NSD it explains held-out variance in vein-adjacent voxels that the scalar correction leaves, and the directed graph outperforms both the undirected and the random graph.

Why not just…?

Why not just regress out anatomical gain, as Kurzawski et al. did? That correction rescales each voxel by a constant. Rescaling changes JNE by the same constant, so it addresses gain but cannot touch pooling of differently tuned upstream sources or output saturation. Pooling is the mechanism most likely to fake stimulus-selective nonlinearity. In a vein voxel draining both face- and scene-preferring tissue, the local slope flips with image category, which is exactly the JNE-SS signature. The simulation stage measures directly how much planted vascular JNE survives scalar regression. If the answer is "almost none," this alternative wins and we will say so.

Why not just normalize JNE by mean slope, or drop voxels near veins? This is the real threat, and the reason the gain mechanism is separated out. If the hierarchy vanishes under gain normalization alone, the expensive model is unnecessary for the headline question. Masking vein-adjacent voxels removes the worst cases but is blind to tangential and deep-to-surface smearing into voxels classed as parenchymal. Vu et al.'s reversal of PPA lateralization suggests such contamination reaches well-localized regions. We run both cheap fixes and report the hierarchy after each, at matched voxel reliability, before claiming anything for the graph model.

Why not just let a flexible spatial network learn cross-voxel mixing without anatomy? A 3D graph or state-space model in the style of ViG3D-UNet or Swin-UMamba would likely predict better. The undirected-graph ablation stands in for this alternative. But unconstrained mixing has no reason to place the nonlinearity in $s_u$ rather than in the mixing. The resulting attribution would be arbitrary, which defeats the purpose. The directed-versus-random-graph comparison is what shows that anatomy carries information.

Why not just use spin-echo acquisitions, which suppress large veins physically? NSD is gradient-echo and cannot be reacquired. Per the cortical-depth paper, spin-echo also forced 1.5 mm rather than 1.0 mm voxels, with added partial-volume blurring. Our approach applies to existing large datasets. Where spin-echo data exist, they become a validation target rather than a competitor.

Decisive Experiments

The first decisive check is simulation recovery. Across phantoms with planted linear neural tuning but strong venous pooling, the naive JNE should report a spurious V1-to-PPA-like hierarchy, and the drainage-aware neural JNE should report it flat. In phantoms with planted neural nonlinearity, both should report it. Failure in either direction falsifies the method.

The second check is the NSD verdict itself. We report the ratio of PPA, FFA and EBA JNE to V1 JNE under the naive model, the scalar correction, gain normalization, and the drainage-aware model, with subject-level confidence intervals. Either outcome is informative. If the hierarchy survives, it is the first vascular-controlled support for Gao et al.'s neural interpretation. If it collapses, a widely usable interpretability result needs revision.

The third check is internal consistency. In V1, the learned upstream weights should make vein-adjacent voxels' recovered neural receptive fields narrower and single-peaked relative to their raw fits. Correcting along the graph should also shrink the horizontal–vertical meridian asymmetry at least as much as the 49%-to-25% reduction Kurzawski et al. obtained, which gives an anatomical check independent of the deep-network features.

The fourth check is an out-of-sample test of PPA lateralization. Vu et al. found right-dominant PPA turning left-dominant after vein correction in four subjects. If our neural components reproduce that shift across NSD's eight subjects without using phase data, the drainage graph is capturing real vascular structure rather than fitting noise.