When we look at a visual scene, our brains do not instantly absorb a smooth, uniform heatmap of important regions. Instead, human vision is an active, sequential process. Our eyes execute a rapid series of ballistic jumps (saccades) punctuated by brief pauses (fixations).
For decades, computer vision models treated visual attention as a static density map, aggregating gaze data over time into a single blurred image of highlights. However, this temporal aggregation introduces a significant blind spot: it collapses the chronological journey of human curiosity into a static destination.
A growing body of research focuses on temporal scanpath prediction and dynamic gaze modeling. By shifting from where humans look on average to the exact sequence and duration of their eye movements, these models capture the underlying cognitive grammar of visual exploration.
The Limitations of Static Saliency
Traditional saliency models excel at identifying which pixels attract attention over an extended viewing period, but they are blind to the trajectory of exploration. As noted by Marc Assens et al. in SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes, static heatmaps omit the sequence and duration of fixations. This omission prevents systems from understanding how a viewer navigates immersive media, such as 360-degree environments, where the order of visual discovery dictates the narrative experience.
Collapsing the temporal dimension also discards the hierarchical structure of scene understanding. Kiwon Yun et al. demonstrate in Studying Relationships between Human Gaze, Description, and Computer Vision that the chronological order of eye movements encodes how humans parse a scene's narrative. An initial glance might register a dominant foreground object, while subsequent fixations contextualize it against the background.
Furthermore, human attention is highly dynamic and shifts across distinct phases during a single viewing session. Heiko H. Schütt et al. demonstrate in Predicting the Fixation Density Over Time that the predictive power of standard saliency models decays as a trial progresses. Viewers rapidly transition from an early, reflexive exploration phase driven by low-level visual features to a deliberate, task-driven equilibrium state. Static models flatten these distinct cognitive phases into a single average.
Static Saliency Map: [ Image ] ---> [ Aggregated Heatmap ] (Time is lost)
Dynamic Scanpath: [ Image ] ---> [ Fixation 1 ] ---> [ Fixation 2 ] ---> [ Fixation 3 ]
The Grammar of Visual Scanpaths
Why does the sequence matter? Human eye movements follow a predictable logic based on the semantic context of the environment. Dong Xu et al. show in Semantically-Based Human Scanpath Estimation with HMMs that human scanpaths exhibit a "category-specific grammar." For instance, when viewing a portrait, the gaze predictably cycles between the eyes, nose, and mouth; when viewing a landscape, it traces the horizon or paths of perspective.
Modeling this chronological trajectory does not just yield more realistic simulations; it also improves static predictions. Single-author work by Bahar Aydemir in TempSAL - Uncovering Temporal Information for Deep Saliency Prediction demonstrates that partitioning a viewing session into discrete sequential intervals serves as a powerful regularizer. By learning the step-by-step trajectory of attention, models can better infer the final, aggregate importance of objects based on the order in which they are prioritized.
This temporal ordering is also a strong proxy for human importance rankings. In complex scenes, the order of initial fixations reveals what objects are immediately salient versus what objects are secondary. As argued by Avishek Siris et al. in Inferring Attention Shifts for Salient Instance Ranking, the chronological sequence of early eye movements serves as a more accurate metric for ranking object importance than the total cumulative duration spent looking at them.
Computational Frameworks for Sequential Gaze
To transition from static density mapping to sequential path generation, researchers leverage architectures capable of handling time-series dependencies and stochastic transitions.
Probabilistic and Markovian Models
Because eye movements transition from one discrete region of interest to another, Hidden Markov Models (HMMs) are widely used to capture these dynamics. Amin Haji Abolhassani et al. utilize this property in Visual Task Inference Using Hidden Markov Models to address the "Inverse Yarbus" problem. By analyzing the sequential transitions between points of interest, their system infers what task a user is performing (e.g., searching for an object versus judging a scene's composition) purely from the chronological flow of their gaze.
Generative Adversarial Networks
Human scanpaths are inherently stochastic; two people looking at the same image will not follow the exact same pixel-by-pixel path, even if their high-level strategies match. To capture this natural variation without falling into the "center-bias" trap of standard supervised learning, Marc Assens et al. introduce PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks. By training a conditional GAN to output sequences of coordinates, the model generates realistic, human-like visual trajectories that preserve the biological constraints of saccadic movement.
Diagnostic and Clinical Applications
Because the temporal rhythm of gaze reflects deep cognitive processing, sequential fixation analysis has emerged as a powerful non-invasive biomarker for psychological profiles and neurological conditions.
- Autism Spectrum Disorder (ASD) Screening: In single-author research, Shi Chen demonstrates in Attention-Based Autism Spectrum Disorder Screening With Privileged Modality that the chronological sequence of visual fixations provides significantly more diagnostic information for behavioral screening than static attention heatmaps. The specific order in which an individual explores social cues reveals underlying cognitive patterns that aggregate data completely flattens.
- Psychiatric Disambiguation: Differentiating between conditions with overlapping clinical symptoms, such as Bipolar Disorder and Major Depressive Disorder, is notoriously difficult. Jonathan Chung 0001 et al. address this in Learning Differences Between Visual Scanning Patterns Can Disambiguate Bipolar and Unipolar Patients by feeding sequences of visual fixations into Recurrent Neural Networks (RNNs). The network isolates subtle spatio-temporal signatures in how patients scan emotional faces, providing an objective metric for clinical diagnosis.
- Personality Trait Prediction: The temporal rhythm of our eyes can even betray our personalities. Sabrina Hoppe et al. demonstrate in Eye Movements During Everyday Behavior Predict Personality Traits that representing sequential eye movement patterns as n-grams allows machine learning models to predict traits like neuroticism and extraversion. The specific order and transitions of saccades and fixations during everyday tasks encode individual psychological profiles more reliably than isolated, aggregate statistics.
By treating vision as a dynamic trajectory rather than a static state, temporal gaze modeling bridges the gap between computer vision and cognitive science, unlocking a deeper understanding of how humans actively perceive the world.