Physics-Informed Dynamics for Visual Scanpath Prediction

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 114 papers26 conceptsfrozen 2026-10-04

Traditional computer vision models treat visual attention as a static heatmap—a "saliency map" that highlights which parts of an image are interesting. However, human vision is inherently temporal and physical. We do not perceive an entire scene at once; instead, our eyes traverse a "scanpath," a sequence of fixations and rapid movements (saccades) constrained by the mechanics of our biology.

Recent research has shifted from predicting static maps to modeling the Focus of Attention (FOA) as a physical mass moving through a dynamic field of forces. By treating gaze as a dynamical system rather than a sequence of discrete classification labels, researchers are creating models that are more biologically plausible, faster, and more accurate.

The Problem with Discrete Jumps

Most deep learning models for attention treat an image as a grid of patches. To predict where a human looks next, these models often use a "winner-take-all" approach, jumping from one high-saliency patch to another.

As Alessandro Betti, Marco Gori, and Stefano Melacci argue in Local propagation of visual stimuli in focus of attention, this results in abrupt, discontinuous jumps that fail to capture the fluid nature of human eye movements. Humans exhibit "smooth pursuit" and trajectories influenced by momentum and physical constraints. A model that simply jumps between pixels ignores the temporal continuity required to simulate realistic vision.

Attention as a Point Mass

To solve this, researchers like Dario Zanca and Marco Gori propose treating the focus of attention as a point mass governed by Newtonian mechanics. In this framework, the gaze point has position, velocity, and acceleration. It moves through a "potential field" where visual features act like gravitational masses, pulling the gaze toward them.

In Visual Attention driven by Convolutional Features, the authors demonstrate that high-level features from a standard CNN can serve as these gravitational pulls. By applying second-order differential equations with damping terms (to simulate the "friction" or energy loss in eye movement), the model generates continuous trajectories that mimic human physiological patterns. This "gravitational" approach allows the model to integrate multiple competing influences—like image boundaries and visual gradients—into a single, smooth motion model.

Integrating Multimodal Forces

The physics-informed approach is naturally extensible. If gaze is a particle moved by forces, we can simply add new "forces" to the equation to account for different stimuli.

In EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics, Xiaochuan Liu and colleagues introduce a framework where gaze is influenced by both visual saliency and "semantic attraction" from synchronized audio. When a narrator mentions a specific object, that object’s "gravitational pull" increases in the model, shifting the gaze trajectory toward it in a way that remains physically plausible and temporally consistent.

Continuous Regression over Discrete Grids

A key technical shift in this research direction is the move away from discrete spatial bins. Instead of predicting which "cell" in a grid a person is looking at, newer architectures like Gazeformer and LogitGaze-Med use continuous coordinate regression.

Why It Works: Biological Plausibility

The primary reason these models outperform traditional deep networks in scanpath prediction is biological plausibility. As shown in Toward Improving the Evaluation of Visual Attention Models, unsupervised gravitational models can produce trajectories so realistic that even experts struggle to distinguish them from real human eye movements.

By incorporating physical principles like acceleration and damping, these models inherently respect the limitations of the human eye. They don't just predict where to look; they simulate the process of looking. This shift from static "seeing" to dynamic "observing" represents a new foundation for computer vision, moving the field toward agents that "learn to see" by interacting with continuous environments rather than processing disconnected snapshots (Deep Learning to See - Towards New Foundations of Computer Vision).

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_physics_informed_dynamics_for_visual_sca_82af681a,
  title = {Physics-Informed Dynamics for Visual Scanpath Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/physics-informed-dynamics-for-visual-scanpath-prediction/rdr_82af681af01b4df2a334e96b85015b45},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_physics_informed_dynamics_for_visual_sca_82af681a_v1,
  title = {Physics-Informed Dynamics for Visual Scanpath Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/physics-informed-dynamics-for-visual-scanpath-prediction/rdr_82af681af01b4df2a334e96b85015b45/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}