Traditional computer vision models treat visual attention as a static heatmap—a "saliency map" that highlights which parts of an image are interesting. However, human vision is inherently temporal and physical. We do not perceive an entire scene at once; instead, our eyes traverse a "scanpath," a sequence of fixations and rapid movements (saccades) constrained by the mechanics of our biology.
Recent research has shifted from predicting static maps to modeling the Focus of Attention (FOA) as a physical mass moving through a dynamic field of forces. By treating gaze as a dynamical system rather than a sequence of discrete classification labels, researchers are creating models that are more biologically plausible, faster, and more accurate.
The Problem with Discrete Jumps
Most deep learning models for attention treat an image as a grid of patches. To predict where a human looks next, these models often use a "winner-take-all" approach, jumping from one high-saliency patch to another.
As Alessandro Betti, Marco Gori, and Stefano Melacci argue in Local propagation of visual stimuli in focus of attention, this results in abrupt, discontinuous jumps that fail to capture the fluid nature of human eye movements. Humans exhibit "smooth pursuit" and trajectories influenced by momentum and physical constraints. A model that simply jumps between pixels ignores the temporal continuity required to simulate realistic vision.
Attention as a Point Mass
To solve this, researchers like Dario Zanca and Marco Gori propose treating the focus of attention as a point mass governed by Newtonian mechanics. In this framework, the gaze point has position, velocity, and acceleration. It moves through a "potential field" where visual features act like gravitational masses, pulling the gaze toward them.
In Visual Attention driven by Convolutional Features, the authors demonstrate that high-level features from a standard CNN can serve as these gravitational pulls. By applying second-order differential equations with damping terms (to simulate the "friction" or energy loss in eye movement), the model generates continuous trajectories that mimic human physiological patterns. This "gravitational" approach allows the model to integrate multiple competing influences—like image boundaries and visual gradients—into a single, smooth motion model.
Integrating Multimodal Forces
The physics-informed approach is naturally extensible. If gaze is a particle moved by forces, we can simply add new "forces" to the equation to account for different stimuli.
In EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics, Xiaochuan Liu and colleagues introduce a framework where gaze is influenced by both visual saliency and "semantic attraction" from synchronized audio. When a narrator mentions a specific object, that object’s "gravitational pull" increases in the model, shifting the gaze trajectory toward it in a way that remains physically plausible and temporally consistent.
Continuous Regression over Discrete Grids
A key technical shift in this research direction is the move away from discrete spatial bins. Instead of predicting which "cell" in a grid a person is looking at, newer architectures like Gazeformer and LogitGaze-Med use continuous coordinate regression.
- Gazeformer, developed by Sounak Mondal and Gregory J. Zelinsky, replaces fixed object detectors with natural language embeddings and regresses continuous spatial coordinates and durations. This captures the inherent variability of human attention better than discrete patches (Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention).
- LogitGaze-Med, by Dmitry Lvov and Ilya Pershin, applies this to medical imaging. By predicting continuous dwell durations and paths for radiologists examining X-rays, they found that these realistic synthetic scanpaths actually help train better downstream AI models for pathology detection (From Human Attention to Diagnosis).
Why It Works: Biological Plausibility
The primary reason these models outperform traditional deep networks in scanpath prediction is biological plausibility. As shown in Toward Improving the Evaluation of Visual Attention Models, unsupervised gravitational models can produce trajectories so realistic that even experts struggle to distinguish them from real human eye movements.
By incorporating physical principles like acceleration and damping, these models inherently respect the limitations of the human eye. They don't just predict where to look; they simulate the process of looking. This shift from static "seeing" to dynamic "observing" represents a new foundation for computer vision, moving the field toward agents that "learn to see" by interacting with continuous environments rather than processing disconnected snapshots (Deep Learning to See - Towards New Foundations of Computer Vision).