Do Cognitive Priors in Scanpath Models Wash Out With Data?

Models that predict where a person will look next in an image fall into two camps. Deep networks learn everything from data. Cognitive models hand-code known facts about eye movements: the eye tends to keep moving in the same direction, it avoids places it just visited, and it drifts horizontally. On the small eye-tracking datasets the field uses, the camps disagree about what matters. The disagreement has never been tested against the one variable that settles such arguments elsewhere in machine learning, which is how much training data the model sees. We propose to measure learning curves for scanpath prediction, meaning how accuracy grows as the model sees more recorded eye movements. We would do this separately for the part of the model that reads the image and the part that tracks the history of previous fixations, and ask which hand-coded cognitive rules a generic sequence model can absorb from data and at what data size.

Our central bet is specific to gaze data. Scanpath data can grow along two different axes: more images, or more viewers per image. We predict that image-driven attention improves mainly with more distinct images. We predict that history-driven dynamics, such as momentum, inhibition of return and the shift from early reflexive viewing to later exploration, are largely independent of image content. If that holds, history dynamics improve with the total number of scanpaths pooled across every available corpus, so hand-coded oculomotor rules should become redundant at a much smaller data size than image-level ones. Confirming or refuting this tells the field whether its next gain comes from collecting more images, more observers, or better cognitive modeling.

Core Claim

The claim is that the value of each cognitive prior shrinks with data at a rate set by the axis that carries the relevant information. The value of a prior is its information gain in bits per fixation. The priors we test are those introduced in What Moves the Eyes: time-dependent temperature, saccadic momentum and cardinal bias. We also test the working-memory inhibition built into IOR-ROI LSTM.

Several papers motivate the question. DeepGaze III splits next-fixation prediction into an image term worth 1.04 bit/fixation and a history term worth 0.50 bit/fixation, but it trains its history network on MIT1003 alone. What Moves the Eyes shows that hand-coding three mechanisms doubles the variance the mechanistic SceneWalk model explains, from 35% to 70%. It never asks whether a deep model trained on more data would find those mechanisms without help. TPP-Gaze models gaze as a sequence of timed events and shows that return fixations emerge without being hard-coded. It evaluates each dataset separately and has no scaling analysis.

One paper constrains how we read the results. Scaling Laws vs Model Architectures found that rigid inductive biases flatten scaling curves in text; for example, Performer's exponent is 0.25 against 0.54 for the vanilla Transformer. We therefore cannot assume that priors which help at small scale remain harmless at larger scale. Generating QM1B is the strongest example of scale replacing hand-built physics. We ask whether the same happens for oculomotor "physics". Our answer can differ from that of QM1B, because eye movements carry real biological constraints that may not be recoverable from a few hundred thousand fixations.

Research Program

Model family. We hold the image pathway fixed across all variants: a DenseNet-201 priority map pretrained on SALICON, as in DeepGaze III. Only the history model varies, and it comes in three versions.

  • Generic model. A causal Transformer, i.e. a sequence model that only sees past events. Each input event is one past fixation, described by its coordinates in degrees of visual angle, its duration and its index. It outputs a spatial log-density over the image grid that is added to the priority map, and a log-normal mixture over the next fixation's duration, following TPP-Gaze.
  • Prior-augmented model. The same Transformer plus the SceneWalk-X mechanisms and an inhibition-of-return term. These enter as additive log-density terms, each multiplied by a learnable gate.
  • Priors only. The priors with no Transformer, which gives SceneWalk-X on a deep priority map.

Corpus. We pool the free-viewing corpora in the file: MIT1003, CAT2000 training, OSIE, COCO-FreeView, NUSEF, FiFa and the Potsdam DAEMONS set. All coordinates are converted to degrees of visual angle, and a learned dataset embedding absorbs differences in eye tracker and viewing duration. The resulting corpus is larger than any single training set used by DeepGaze III or IOR-ROI LSTM.

Scaling fits. We subsample along two independent axes: $I$, the number of distinct images, and $S$, the number of observers per image. For each variant $m$ we fit held-out loss in bits per fixation as

$$L_m(I,S) = L_{\infty,m} + A_m I^{-\alpha_m} + B_m (IS)^{-\beta_m},$$

where $L_{\infty,m}$ is the irreducible loss. The $I$ term captures learning of image content. The $IS$ term, driven by the total scanpath count, captures learning of history dynamics. $A_m$, $B_m$, $\alpha_m$ and $\beta_m$ are fitted constants. The power-law form follows the learning-curve convention of Banko et al. and Tay et al., and we test it against a saturating alternative.

Readouts. For each prior we report the gap $\Delta_m(N) = L_{\text{generic}}(N) - L_{\text{prior}}(N)$ against scanpath count $N$, together with its learned gate weight. The estimated crossover data size is the main result, and we will report it even when it lies beyond the available data. As a secondary question we test whether conditioning on continuous fixation duration improves next-fixation spatial likelihood, and whether that benefit grows with scale. The file identifies this as open for TPP-Gaze.

Baselines

Center bias plus static saliency. The naive reference samples the next fixation from DeepGaze III's image-only priority map, ignoring history. It fixes the floor that every history model must beat, and at full scale it should trail by about the 0.50 bit/fixation that DeepGaze III attributes to history.

DeepGaze III. This is the strongest existing method. It is reported at 2.442 bit/fixation on MIT1003, 2.248 on MIT300 and 3.064 on CAT2000 in DeepGaze III. We retrain it on each identical data subset with the same backbone, the same number of optimizer steps and the same hyperparameter search budget as our variants. At full data our generic model must match or exceed it on held-out MIT300-style evaluation. If the generic model loses at every scale, the architecture rather than the data is the bottleneck, and we would say so.

SceneWalk-X. This is the priors-only arm, anchored by the 70% explained variance reported on MIT1003 in What Moves the Eyes.

TPP-Gaze. We rerun it on the same subsets because it is the strongest duration-aware model. The file reports no numeric MultiMatch value for it on COCO-FreeView, so on our splits it serves as a reference, not a target number.

Key ablations. Two ablations isolate the main ingredient. The first removes one prior at a time from the prior-augmented model. The second varies $I$ and $S$ while holding the total fixation count fixed, which is the only way to separate the two scaling axes. All models are evaluated by fixation-by-fixation conditional log-likelihood, AUC and NSS (normalized scanpath salience, the average map value at true fixations), following State-of-the-Art in Human Scanpath Prediction. That protocol avoids the tuning sensitivity of whole-scanpath similarity metrics.

Why not just…?

Why not just keep the priors behind learnable gates and let training switch them off? This is a real competitor for practical use, and we build it in as the prior-augmented model. The gates alone do not answer the scientific question. A gate can stay open because the prior is useful, or because the Transformer never had enough data to learn the prior's function. Only the paired generic-model curve distinguishes the two cases. If gated models dominate at every scale, the practical recommendation becomes "always include gated priors", and we will report that.

Why not just sample from a strong static saliency map with an inhibition mask? DeepGaze III's 0.50 bit/fixation history gain and the decline over viewing time documented in Predicting the Fixation Density Over Time argue that a static map misses structure. This option is also our naive baseline, so its gap to the other models is measured at every data size, not assumed.

Why not just pretrain on unlimited SceneWalk-X simulations instead of collecting real data? This is the most serious threat, because simulated scanpaths turn the priors into data, and that could make the scaling question moot. We add it as an experimental arm: pretrain the generic model on simulated scanpaths, then fine-tune on real subsets. If this arm matches the prior-augmented model everywhere, the priors are fully learnable from data but real data is too scarce to teach them. If it also stalls short of the prior-augmented model, the simulator lacks structure that human data contains.

Why not just train DeepGaze III on the pooled corpus? We do this in the DeepGaze III baseline. Its fixed window of four past fixations may cap the history gain at large scale. Comparing its $\beta$ exponent with the generic model's tests whether that cap exists.

Decisive Checks

The central prediction is falsified in either of two cases. The first is if the history exponent $\beta$ is no larger than the image exponent $\alpha$. The second is if the observer axis $S$, at fixed $IS$, fails to reduce the history-term error. A cleaner test of image independence trains the history model on goal-directed COCO-Search18 scanpaths, about 300,000 search fixations (COCO-Search18), and transfers it frozen to free viewing. If history dynamics are largely image-independent, frozen transfer should recover most of the 0.50 bit/fixation history gain. If it recovers little, the dynamics are task- or image-bound, and pooling across datasets cannot replace the cognitive rules.