Probabilistic Modeling of Human Gaze and Scanpaths

28 concepts23 papers1 proposal

When we look at an image, our eyes do not track a deterministic, universally shared path. Visual exploration is inherently stochastic, noisy, and deeply subjective. If you show the same photograph to ten different people, they will focus on different regions, follow different sequences of fixations, and hold entirely different cognitive motivations for doing so.

Yet, for years, computational models treated visual attention as a deterministic regression task. Traditional saliency models aimed to predict either a single consensus heatmap or a precise coordinate for the next gaze point. This deterministic framing creates a fundamental mismatch with human biology: it forces models to collapse a rich distribution of human behaviors into a single, blurry "average" observer, often falling into the trap of center-bias.

To fix this, a major research shift has reframed visual attention. Instead of asking where the average person looks, modern frameworks treat gaze as a spatial probability distribution and scanpaths as stochastic trajectories that can be sampled from generative models.

From Coordinates to Spatial Distributions

The core insight of probabilistic gaze modeling is that human attention is better represented as a continuous landscape of likelihoods rather than a fixed set of pixel coordinates.

To capture this mathematically, models had to move away from standard mean-squared error regression losses, which penalize deviations from an arbitrary average. Instead, the authors of End-to-End Saliency Mapping via Probability Distribution Prediction (Saumya Jetley et al.) proposed training deep fully-convolutional networks using probability distance metrics, such as the Bhattacharyya distance. By treating the ground-truth fixations as a spatial distribution, the network learns to predict the entire topographical landscape of visual interest.

Deterministic Regression:   Image ──> [ Deep Network ] ──> Single Average Coordinate (or Blurry Heatmap)
Probabilistic Modeling:     Image ──> [ Deep Network ] ──> Spatial Probability Distribution P(x, y)

This perspective allows for a much more rigorous mathematical treatment of attention. As Matthias Kuemmerer et al. detailed in Predicting Visual Fixations, modeling attention as a full probability distribution over image coordinates allows researchers to leverage information-theoretic metrics. This framework makes it possible to quantify exactly how much information an image provides about gaze variance, moving the field past "black box" heuristics.

When we accept that gaze is a distribution, we can also predict when humans will actually agree. Anna Volokitin et al. demonstrated in Predicting When Saliency Maps are Accurate and Eye Fixations Consistent that inter-observer consistency is itself a predictable attribute of an image, heavily driven by the semantic categories of the objects present. Some images naturally command a tight consensus, while others trigger highly divergent viewing patterns.

Generative Scanpaths and Stochastic Sampling

While static heatmaps tell us where groups of people look over time, they throw away the temporal sequence of vision—the scanpath. Humans explore environments via a series of rapid eye movements (saccades) interspersed with brief pauses (fixations). Because different observers take entirely different trajectories across the same image, standard supervised learning fails; it tries to force a single trajectory output, resulting in unrealistic, robotic paths.

To generate realistic human eye movements, researchers have turned to generative architectures that model the distribution of possible paths. Instead of outputting a single next step, models can output the parameters of a mixture distribution. For example, in Pathformer3D: A 3D Scanpath Transformer for $360^{\circ }$ Images, Rong Quan et al. designed a transformer that outputs the parameters for a mixture of 3D Gaussian distributions. To simulate a specific virtual subject, you simply sample a path from this evolving distribution.

Supervised Path Prediction:  Step N ──> [ Model ] ──> Fixed Step N+1 (Collapses to average)
Generative Path Sampling:    Step N ──> [ Model ] ──> Parameters of Mixture Gaussian ──> [ Sample ] ──> Diverse Step N+1

Several generative paradigms have been successfully applied to this problem:

Modeling Subjectivity and Latent Intent

Why do human scanpaths diverge? Often, the variance is caused by differing internal states, tasks, or cognitive backgrounds.

To capture this inherent subjectivity, models must represent the latent factors guiding attention. Jing Zhang et al. addressed this in Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction and Energy-Based Generative Cooperative Saliency Prediction by pairing a fast Latent Variable Model (LVM) with a refining Energy-Based Model (EBM). This allows the network to produce diverse, plausible saliency maps for a single image, yielding meaningful uncertainty maps that highlight where human observers are likely to disagree. To train models to accept this divergence, the authors also introduced a simple "random sampling" strategy in A General Divergence Modeling Strategy for Salient Object Detection, forcing networks to learn the full distribution of diverse human annotations rather than a single consensus.

This subjectivity becomes critical in expert domains. Ngan Le demonstrates in Modeling radiologists’ cognitive processes using a digital gaze twin to enhance radiology training that expert radiologists navigate the same chest X-ray using highly distinct but equally valid visual search strategies. A deterministic model would fail to capture these diverse clinical paths. Furthermore, even when two observers look at the exact same object, they may do so for entirely different reasons. To bridge this gap, Xianyu Chen et al. developed GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths, aligning probabilistic scanpath generation with natural language rationales to explain the underlying intent behind the eye movements.

Robustness to Noise in Downstream Tasks

Modeling gaze as a probability distribution is not just a theoretical victory; it provides crucial engineering advantages in real-world applications by absorbing measurement noise and human behavioral quirks.

In first-person (egocentric) video analysis, eye-tracking data is notoriously unstable due to rapid head movements and hardware calibration drift. In In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video and In the Eye of the Beholder: Gaze and Actions in First Person Video, Yin Li et al. demonstrated that treating gaze as a probabilistic latent variable rather than a fixed coordinate allows a system to gracefully handle measurement noise and rapid eye jumps. This probabilistic smoothing directly improves downstream performance in tasks like action recognition.

Point-based Gaze:       [ Noisy Coordinate (X,Y) ] ──> High Sensitivity to Calibration Drift/Jitter
Probabilistic Gaze:     [ Spatial Likelihood Map ] ──> Smooths out Jitter, Retains Contextual Focus

This resilience to noise also benefits computer vision tasks like object segmentation. Human fixations naturally wander around an object rather than clicking precisely on its boundaries. By treating fixations as a noisy probability distribution, the framework proposed by Gongyang Li et al. in Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation successfully navigates the ambiguity of eye-tracking data to isolate clean object boundaries.

Finally, understanding this stochasticity is essential when trying to manipulate user attention. As Yujia Wang et al. noted in Target Scanpath-Guided 360-Degree Image Enhancement, because individual gaze patterns vary so widely in immersive 360-degree environments, creating universal image modifications that reliably guide a diverse population along a specific target path remains a complex, open challenge.

By moving from rigid coordinates to flexible distributions, probabilistic gaze modeling provides the mathematical tools necessary to simulate, explain, and utilize the true, stochastic nature of human vision.

Key Foundational Work

End-to-End Saliency Mapping via Probability Distribution Prediction (2016) — contributed 1 concept elements to this direction.
Predicting When Saliency Maps are Accurate and Eye Fixations Consistent (2016) — contributed 1 concept elements to this direction.
How Does Image Content Affect the Added Value of Visual Attention in Objective Image Quality Assessment? (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_probabilistic_gaze_modeling_c8326f18,
  title = {probabilistic gaze modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probabilistic-gaze-modeling/rdr_c8326f1861e045c0ae7881abbb26527e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_probabilistic_gaze_modeling_c8326f18_v1,
  title = {probabilistic gaze modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probabilistic-gaze-modeling/rdr_c8326f1861e045c0ae7881abbb26527e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake