Posture Gain Fields in Retinotopic Neurons During Free Viewing

Neurons in visual cortex describe the world in eye-centered coordinates: a cell responds to a feature at a fixed spot on the retina, wherever the eyes point. Some well-controlled laboratory studies, though, show that the strength of those responses depends on where the eyes point relative to the head and body. This multiplicative scaling is called a gain field. It is the ingredient that classic models use to convert eye-centered information into head- or body-centered information without ever moving a receptive field. This note proposes a direct test of whether gain fields shape visual responses during natural vision.

The test fits encoding models to a very large recording of macaque visual cortex made while animals freely explored natural images. An encoding model is a predictor of each neuron's firing from the image it sees. The candidate model has a strict factorized form: an eye-centered receptive field multiplied by a smooth, low-dimensional function of gaze angle. We ask whether this form predicts held-out responses better than a purely eye-centered model and better than an unconstrained network that is simply given gaze as an extra input. We also ask whether the fitted gains peak at the straight-ahead direction, as laboratory work on a "straight-ahead bias" predicts. Free viewing breaks the link between retinal position and gaze direction on every fixation. It is therefore a natural decoupling experiment that has so far been analyzed in only one coordinate frame.

Core Claim

Two recent findings pull in different directions, and each leaves a gap for the other. Feature-selective responses in macaque visual cortex follow eye movements during natural vision recorded 883 hours and 4.6 million fixations across V1, V2, V4 and anterior inferior temporal cortex (AIT). It found that 95.7% of feature-selective neurons respond according to eye position, with gaze-locked precision down to 0.25° even in AIT. That analysis asked where receptive fields sit. It did not ask whether response amplitude scales with gaze angle.

Three laboratory studies do show such scaling:

All three used flashed checkerboards or wedges at a few static gaze positions.

Our claim is that the natural-vision result and the laboratory results are compatible, and that the factorized model is how to show it. Receptive-field position stays retinotopic, as Xiao et al. found, while amplitude carries a structured posture gain. The candidate model for neuron $i$ is

$$r_i(t) = g_i(\mathbf{e}_t)\, f_i\big(I_t;\ \mathbf{e}_t\big) + b_i ,$$

where the terms are defined as follows:

  • $r_i(t)$ is the neuron's response during fixation $t$.
  • $I_t$ is the image and $\mathbf{e}_t$ is the two-dimensional gaze angle relative to the head.
  • $f_i$ is a retinotopic image model: deep-network features read out at a receptive-field location defined relative to the fovea.
  • $g_i$ is a positive gain function of gaze. It is either planar or a small radial basis centered at candidate reference points.
  • $b_i$ is a baseline.

The theory at stake comes from Hierarchical Transformation of Space in the Visual System, which showed that networks trained to compute head-centered position from retinal position plus eye position develop exactly this retinotopic-times-gain structure. That work used synthetic Gaussian blobs. We test the structure on natural images and natural saccades.

Research Program

Data preparation. We use the public free-viewing recordings from Xiao et al. For each fixation we record the image, gaze position, and fixation onset. We pair each of these with the preceding saccade vector, fixation duration, ordinal position within the trial, and the location of the next saccade target. These covariates are what make the analysis honest. Gaze angle is correlated with saccade history and with where the animal is about to look. An apparent gain field could therefore be adaptation, a saccade transient, or pre-saccadic attention.

Model fitting. We fit $f_i$ with a frozen image backbone and a per-neuron spatial readout, following the retinotopic-readout logic of Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe. That work maps brain coordinates to visual-field coordinates for static-image fMRI encoding. We borrow its idea of a readout constrained to retinotopic positions, apply it to single units, and add a gaze-dependent amplitude term that it does not model. The gain $g_i$ is fit jointly under cross-validation over images rather than over fixations. This split prevents the same scene from leaking between training and testing.

Analyses of the fitted gains. Three analyses follow from the fitted models.

  1. We estimate, per area, the fraction of neurons whose held-out prediction improves significantly with the gain term. This is the natural-vision counterpart of Merriam et al.'s roughly 20% of voxels.
  2. We ask whether gain peaks cluster at 0° gaze-in-head. That would carry the straight-ahead bias into natural vision. We also ask whether the gains are heterogeneous in the way Zipser and Andersen predict, which would support population-level coordinate transformation rather than a uniform arousal effect.
  3. We train a linear decoder of gaze angle from population responses to test whether the gains carry usable posture information. We then test whether that decodability survives across areas up to AIT.

Limits of the design. Because the animals' heads are, as far as we know from the recording setup, fixed relative to the trunk, this design cannot separate head from trunk. That is the open question raised by Cottereau et al. The model instead produces a sharp, quantitative prediction for the follow-up human experiment described below.

Baselines

Naive reference. The simplest reference predicts each neuron's response from its mean rate plus the nuisance covariates: saccade vector, fixation duration, fixation order, and next-target location. It uses no image. Any image-based model must beat it. Any gain-field effect must add to it, rather than being absorbed by it.

Strongest existing method. The strongest existing method is the purely eye-centered encoding model, $f_i$ alone plus the covariates. This is the analysis stance of Xiao et al., upgraded to a deep-feature readout. It is the right competitor because Xiao et al. report that responses depend entirely on eye position. If gaze gains are negligible under natural viewing, this model will match ours. No reported encoding accuracy (correlation or explained variance) for this dataset appears on Lacuna, so we will establish the number ourselves under identical splits. Likewise, no reported $R$ or $R^2$ was found on Lacuna for the RetinaMapper benchmark.

Flexible competitor. The strongest flexible competitor is an unconstrained network that receives gaze angle as extra inputs. Gaze is concatenated to the backbone features and passed through the same readout with a matched parameter budget and hyperparameter search.

Ablations. Two ablations isolate the key ingredient. The first replaces the multiplicative gain with an additive gaze offset, which tests gain against baseline shift. The second is a spatiotopic model whose receptive field is fixed in screen coordinates. It is the direct analog of the retinotopic-versus-spatiotopic contrast in Maps of Visual Space in Human Occipital Cortex Are Retinotopic, Not Spatiotopic.

Fairness and the bar for success. All models share data splits, backbone, covariates, and tuning budget. Beating the baselines means a significant held-out improvement over the eye-centered model in a consistent fraction of neurons per area, established by paired tests across neurons. The factorized model must also match the flexible network within noise while using a gain of only a few parameters.

Why not just…?

Why not just accept the eye-centered model, since 95.7% of neurons already follow the eyes? "Following the eyes" constrains where a receptive field sits. It does not constrain its amplitude. Merriam et al. found fixed phases alongside modulated amplitudes in the same voxels. The eye-centered model is our main baseline, and if it wins, that is a publishable negative result. It would show that laboratory gain fields do not survive natural viewing, which would constrain how the straight-ahead bias should be interpreted.

Why not just feed gaze into a flexible deep network and see if prediction improves? This alternative is a real threat on raw accuracy, and we run it. A flexible network can use gaze to shift receptive fields, reweight features, or model saccade transients. Its improvement alone therefore says nothing about coordinate frames. If it clearly beats the factorized model, gaze acts through something other than gain, and we will report which component it uses by inspecting receptive-field shifts across gaze bins. If the two tie, the low-dimensional gain is the more parsimonious account and yields interpretable peaks.

Why not just run the head-versus-trunk experiment in humans directly? That experiment is the ultimate goal, and we propose it as the follow-up. It is costly and asks a narrower question with flashed probes. The free-viewing analysis first establishes whether gain fields matter under natural behavior at all. It also supplies gain shapes that can be converted into a quantitative prediction for how responses should change when the head is turned on the trunk.

Why not just call it spatial attention to the center? Cottereau et al. argue that the bias is neither endogenous nor exogenous attention, and Bogdanova et al. show it is present before the stimulus appears. In free viewing, the realistic attentional confound is the upcoming saccade target. We include it as a covariate and repeat the analysis on early fixation windows, before pre-saccadic effects build up.

Decisive Checks

The first check is held-out prediction. The gain model must beat the eye-centered model under image-level cross-validation after the saccade, adaptation, and next-target covariates are included. The second is a permutation control: gaze labels shuffled within gaze-matched image sets must abolish the gain improvement. The third is a peak-location test: gain peaks should concentrate near 0° gaze-in-head more than a uniform null over the sampled gaze range would predict. The fourth is a decoding benchmark: population decoding of gaze should reach or exceed the roughly 7° reconstruction accuracy that Merriam et al. obtained from fMRI, since single units should carry at least as much information.

If all four hold, natural-vision retinotopy and laboratory gain fields become a single account. That account gives the head-versus-trunk experiment a concrete prediction: whether the gain peak follows the nose or the sternum when the head turns.