Abstract
Human visual behavior, captured through scanpaths, provides critical insights into cognitive processes and clinical conditions. However, modeling these multivariate time series effectively remains challenging due to spatial misalignment and temporal neglect. We present SpFormer, a Transformer-based framework that employs a fixation-centric alignment strategy and a bio-inspired "meta scanpath" mechanism. SpFormer achieves state-of-the-art performance across multiple benchmarks, including a 100% accuracy rate in identifying Autism Spectrum Disorder (ASD) on the Saliency4ASD dataset.
Introduction
Human visual attention is characterized by scanpaths—sequences of $(x, y)$ coordinates (fixations) and their corresponding durations. Analyzing this data is vital for understanding human perception and aiding clinical diagnoses. Despite the potential of deep learning, existing models struggle with three primary issues:
- Spatial Misalignment: Global feature maps often fail to precisely represent the visual information at a specific gaze point due to neural network downsampling.
- Temporal Neglect: Many methods overlook the chronological sequence and duration of fixations, treating gaze data as static heatmaps.
- Semantic Redundancy: Human gaze often includes "noisy" or redundant fixations that standard Transformers may over-process.
Methodology
SpFormer addresses these hurdles through three architectural innovations:
Fixation-Centric Paradigm
To ensure precise spatial alignment, SpFormer crops small image patches centered exactly on each fixation point. These patches are tokenized, ensuring that the visual features processed by the model correspond directly to what the subject was viewing.
Local Meta Attention
Inspired by the Visual Working Memory (VWM) theory—which suggests humans can only store 3–4 visual items at once—SpFormer utilizes a local attention module. This module generates a "meta scanpath," a condensed version of the original gaze sequence that filters out redundant information.
Progressive Duration Aggregation (PDA)
Fixation duration reflects cognitive load. Rather than treating duration as a simple numerical input, SpFormer uses a progressive decay mechanism to integrate duration information into visual features across Transformer blocks. This helps the model resolve location ambiguity as features become more abstract.
Experimental Results
The model was evaluated on four datasets: Saliency4ASD, a new Collected Dataset (58 subjects), Toddler Age Prediction (TAP), and Visual Perceptual Task (VPT).
Performance Benchmarks
SpFormer established new state-of-the-art (SOTA) results:
- ASD Recognition: Achieved 100% subject-wise accuracy on Saliency4ASD, significantly outperforming previous methods like APM (89%).
- Toddler Age Prediction: Reached 87.5% accuracy, an 11.67% improvement over prior bests.
- Visual Task Prediction: Achieved 97.5% accuracy, outperforming existing methods by over 11%.
Key Insights
Ablation studies highlighted that the fixation-centric alignment alone provided a 15.67% boost in accuracy. Furthermore, incorporating cognitive constraints (the VWM bottleneck) acted as a powerful regularizer, improving the model's ability to interpret human behavior.
Conclusion
SpFormer demonstrates that bio-inspired constraints and precise spatial-temporal alignment are crucial for modeling human gaze. While the model shows sensitivity to certain hyperparameters like the duration decay coefficient, its superior performance across diverse tasks suggests significant potential for clinical and research applications. Future work will explore the model's robustness on lower-frequency eye-tracking data, such as that from standard webcams.
Summary updated