SpFormer: Spatio-Temporal Modeling for Scanpaths with Transformer

Wenqi Zhong, Linzhi Yu, Chen Xia, Junwei Han, Dingwen Zhang

Abstract

Human visual behavior, captured through scanpaths, provides critical insights into cognitive processes and clinical conditions. However, modeling these multivariate time series effectively remains challenging due to spatial misalignment and temporal neglect. We present SpFormer, a Transformer-based framework that employs a fixation-centric alignment strategy and a bio-inspired "meta scanpath" mechanism. SpFormer achieves state-of-the-art performance across multiple benchmarks, including a 100% accuracy rate in identifying Autism Spectrum Disorder (ASD) on the Saliency4ASD dataset.

Introduction

Human visual attention is characterized by scanpaths—sequences of $(x, y)$ coordinates (fixations) and their corresponding durations. Analyzing this data is vital for understanding human perception and aiding clinical diagnoses. Despite the potential of deep learning, existing models struggle with three primary issues:

  1. Spatial Misalignment: Global feature maps often fail to precisely represent the visual information at a specific gaze point due to neural network downsampling.
  2. Temporal Neglect: Many methods overlook the chronological sequence and duration of fixations, treating gaze data as static heatmaps.
  3. Semantic Redundancy: Human gaze often includes "noisy" or redundant fixations that standard Transformers may over-process.

Methodology

SpFormer addresses these hurdles through three architectural innovations:

Fixation-Centric Paradigm

To ensure precise spatial alignment, SpFormer crops small image patches centered exactly on each fixation point. These patches are tokenized, ensuring that the visual features processed by the model correspond directly to what the subject was viewing.

Local Meta Attention

Inspired by the Visual Working Memory (VWM) theory—which suggests humans can only store 3–4 visual items at once—SpFormer utilizes a local attention module. This module generates a "meta scanpath," a condensed version of the original gaze sequence that filters out redundant information.

Figure 1: Comparison between raw scanpaths (top) and the condensed meta scanpaths (bottom) generated by SpFormer. Stars indicate fixation positions, while pink circles represent duration.

Progressive Duration Aggregation (PDA)

Fixation duration reflects cognitive load. Rather than treating duration as a simple numerical input, SpFormer uses a progressive decay mechanism to integrate duration information into visual features across Transformer blocks. This helps the model resolve location ambiguity as features become more abstract.

Experimental Results

The model was evaluated on four datasets: Saliency4ASD, a new Collected Dataset (58 subjects), Toddler Age Prediction (TAP), and Visual Perceptual Task (VPT).

Performance Benchmarks

SpFormer established new state-of-the-art (SOTA) results:

  • ASD Recognition: Achieved 100% subject-wise accuracy on Saliency4ASD, significantly outperforming previous methods like APM (89%).
  • Toddler Age Prediction: Reached 87.5% accuracy, an 11.67% improvement over prior bests.
  • Visual Task Prediction: Achieved 97.5% accuracy, outperforming existing methods by over 11%.

Key Insights

Ablation studies highlighted that the fixation-centric alignment alone provided a 15.67% boost in accuracy. Furthermore, incorporating cognitive constraints (the VWM bottleneck) acted as a powerful regularizer, improving the model's ability to interpret human behavior.

Conclusion

SpFormer demonstrates that bio-inspired constraints and precise spatial-temporal alignment are crucial for modeling human gaze. While the model shows sensitivity to certain hyperparameters like the duration decay coefficient, its superior performance across diverse tasks suggests significant potential for clinical and research applications. Future work will explore the model's robustness on lower-frequency eye-tracking data, such as that from standard webcams.

Summary updated

Abstract

Saccadic scanpath, a data representation of human visual behavior, has received broad interest in multiple domains. Scanpath is a complex eye-tracking data modality that includes the sequences of fixation positions and fixation duration, coupled with image information. However, previous methods usually face the spatial misalignment problem of fixation features and loss of critical temporal data (including temporal correlation and fixation duration). In this study, we propose a Transformer-based scanpath model, SpFormer, to alleviate these problems. First, we propose a fixation-centric paradigm to extract the aligned spatial fixation features and tokenize the scanpaths. Then, according to the visual working memory mechanism, we design a local meta attention to reduce the semantic redundancy of fixations and guide the model to focus on the meta scanpath. Finally, we progressively integrate the duration information and fuse it with the fixation features to solve the problem of ambiguous location with the Transformer block increasing. We conduct extensive experiments on four databases under three tasks. The SpFormer establishes new state-of-the-art results in distinct settings, verifying its flexibility and versatility in practical applications. The code can be obtained from https://github.com/wenqizhong/SpFormer.
BibTeX
@inproceedings{zhong2024spformer,
  author = {Wenqi Zhong and Linzhi Yu and Chen Xia and Junwei Han and Dingwen Zhang},
  title = {{SpFormer: Spatio-Temporal Modeling for Scanpaths with Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28593},
}
Figure