Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation

Swati Jindal, Mohit Yadav, Roberto Manduchi

Introduction

Gaze estimation—the ability of a computer to determine where a person is looking—is a fundamental component of modern human-computer interaction. While significant progress has been made in static image gaze estimation, video-based estimation presents unique challenges. Specifically, video sequences are often plagued by "distractors": irrelevant temporal changes such as facial expressions, mouth movements, or background activity that do not correlate with gaze direction. Furthermore, individual differences in eye physiology and head shape necessitate personalization, which traditionally requires large amounts of user-specific data.

To address these issues, researchers have introduced STAGE (Spatio-Temporal Attention for Gaze Estimation), a framework designed to filter out noise and enable efficient, few-shot personalization.

The STAGE Framework

The STAGE architecture is built on a modular pipeline that processes video frames to extract gaze-relevant features while maintaining temporal consistency.

The STAGE architecture overview, integrating ResNet feature extraction, the Spatial Attention Module (SAM), a Temporal Sequence Model (TSM), and Gaussian Process Layers (GPL).

The framework consists of three primary stages:

  1. Spatial Attention Module (SAM): This module acts as a filter. By comparing consecutive frames, it identifies motion related to the eyes and head while suppressing irrelevant changes. The authors found that a "Hybrid-SAM" approach, which combines dual-frame attention with cross-attention mechanisms, provides the most robust feature extraction.
  2. Temporal Sequence Model (TSM): Features from SAM are passed into a TSM (either an RNN or a Transformer) to capture the long-term dynamics of gaze movement.
  3. Gaussian Process (GP) Personalization: A non-parametric layer is added to the end of the network to learn individual-specific biases. Because GPs are mathematically robust in low-data regimes, they can adjust the general model's output for a specific person using just 3–16 calibration samples.

Key Insights: Gaze as Change

A core innovation of STAGE is treating gaze estimation as a "change-detection" task rather than simple feature extraction. By focusing on the differences between frames, the model naturally prioritizes the movement of the eyes and head.

Visualization of attention maps showing how the model suppresses distractors like background movement and facial expressions while focusing on gaze-relevant features.

As shown in the attention maps, the SAM effectively ignores common distractors. Whether it is a person running in the background or the subject sticking out their tongue, the model maintains its focus on the ocular region. This suppression is critical for maintaining accuracy in unconstrained, real-world environments.

Efficient Personalization and Uncertainty

Personalizing deep learning models usually involves expensive fine-tuning. STAGE bypasses this by using Gaussian Processes (GPs) for "plug-and-play" personalization. This approach offers two major advantages:

  • Few-Shot Learning: The model can be calibrated for a new user with as few as 3 samples, achieving significant accuracy gains (e.g., an additional 0.8° improvement).
  • Uncertainty Estimation: Unlike standard neural networks, GPs provide a mathematical measure of confidence. The authors demonstrated that higher uncertainty correlates with higher prediction error, allowing the system to flag potentially unreliable gaze estimates in real-time.

Performance and Results

STAGE was evaluated across three major benchmarks: Gaze360, EYEDIAP, and EVE. The results demonstrate its superiority:

  • State-of-the-Art Accuracy: On the Gaze360 dataset, STAGE reduced mean angular error by 2.5° compared to previous non-personalized methods.
  • Cross-Dataset Robustness: In challenging cross-dataset scenarios (training on EVE and testing on EyeDiap), the Hybrid-SAM model achieved a 14.28% relative error reduction over baselines.

Conclusion

By combining spatio-temporal attention with the mathematical rigor of Gaussian Processes, STAGE provides a robust solution for video gaze estimation. Its ability to ignore environmental distractors and adapt to new users with minimal data makes it a highly practical framework for the next generation of gaze-aware applications.

Reference: Jindal, S., Yadav, M., & Manduchi, R. (2024). Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation. arXiv:2404.05215v2.

Summary updated

Abstract

Gaze is an essential prompt for analyzing human behavior and attention. Recently, there has been an increasing interest in determining gaze direction from facial videos. However, video gaze estimation faces significant challenges, such as understanding the dynamic evolution of gaze in video sequences, dealing with static backgrounds, and adapting to variations in illumination. To address these challenges, we propose a simple and novel deep learning model designed to estimate gaze from videos, incorporating a specialized attention module. Our method employs a spatial attention mechanism that tracks spatial dynamics within videos. This technique enables accurate gaze direction prediction through a temporal sequence model, adeptly transforming spatial observations into temporal insights, thereby significantly improving gaze estimation accuracy. Additionally, our approach integrates Gaussian processes to include individual-specific traits, facilitating the personalization of our model with just a few labeled samples. Experimental results confirm the efficacy of the proposed approach, demonstrating its success in both within-dataset and cross-dataset settings. Specifically, our proposed approach achieves state-of-the-art performance on the Gaze360 dataset, improving by $2.5^\circ$ without personalization. Further, by personalizing the model with just three samples, we achieved an additional improvement of $0.8^\circ$. The code and pre-trained models are available at \url{https://github.com/jswati31/stage}.
BibTeX
@inproceedings{jindal2024spatio,
  author = {Swati Jindal and Mohit Yadav and Roberto Manduchi},
  title = {{Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00065},
}
Figure