Introduction
Gaze estimation—the ability of a computer to determine where a person is looking—is a fundamental component of modern human-computer interaction. While significant progress has been made in static image gaze estimation, video-based estimation presents unique challenges. Specifically, video sequences are often plagued by "distractors": irrelevant temporal changes such as facial expressions, mouth movements, or background activity that do not correlate with gaze direction. Furthermore, individual differences in eye physiology and head shape necessitate personalization, which traditionally requires large amounts of user-specific data.
To address these issues, researchers have introduced STAGE (Spatio-Temporal Attention for Gaze Estimation), a framework designed to filter out noise and enable efficient, few-shot personalization.
The STAGE Framework
The STAGE architecture is built on a modular pipeline that processes video frames to extract gaze-relevant features while maintaining temporal consistency.
The framework consists of three primary stages:
- Spatial Attention Module (SAM): This module acts as a filter. By comparing consecutive frames, it identifies motion related to the eyes and head while suppressing irrelevant changes. The authors found that a "Hybrid-SAM" approach, which combines dual-frame attention with cross-attention mechanisms, provides the most robust feature extraction.
- Temporal Sequence Model (TSM): Features from SAM are passed into a TSM (either an RNN or a Transformer) to capture the long-term dynamics of gaze movement.
- Gaussian Process (GP) Personalization: A non-parametric layer is added to the end of the network to learn individual-specific biases. Because GPs are mathematically robust in low-data regimes, they can adjust the general model's output for a specific person using just 3–16 calibration samples.
Key Insights: Gaze as Change
A core innovation of STAGE is treating gaze estimation as a "change-detection" task rather than simple feature extraction. By focusing on the differences between frames, the model naturally prioritizes the movement of the eyes and head.
As shown in the attention maps, the SAM effectively ignores common distractors. Whether it is a person running in the background or the subject sticking out their tongue, the model maintains its focus on the ocular region. This suppression is critical for maintaining accuracy in unconstrained, real-world environments.
Efficient Personalization and Uncertainty
Personalizing deep learning models usually involves expensive fine-tuning. STAGE bypasses this by using Gaussian Processes (GPs) for "plug-and-play" personalization. This approach offers two major advantages:
- Few-Shot Learning: The model can be calibrated for a new user with as few as 3 samples, achieving significant accuracy gains (e.g., an additional 0.8° improvement).
- Uncertainty Estimation: Unlike standard neural networks, GPs provide a mathematical measure of confidence. The authors demonstrated that higher uncertainty correlates with higher prediction error, allowing the system to flag potentially unreliable gaze estimates in real-time.
Performance and Results
STAGE was evaluated across three major benchmarks: Gaze360, EYEDIAP, and EVE. The results demonstrate its superiority:
- State-of-the-Art Accuracy: On the Gaze360 dataset, STAGE reduced mean angular error by 2.5° compared to previous non-personalized methods.
- Cross-Dataset Robustness: In challenging cross-dataset scenarios (training on EVE and testing on EyeDiap), the Hybrid-SAM model achieved a 14.28% relative error reduction over baselines.
Conclusion
By combining spatio-temporal attention with the mathematical rigor of Gaussian Processes, STAGE provides a robust solution for video gaze estimation. Its ability to ignore environmental distractors and adapt to new users with minimal data makes it a highly practical framework for the next generation of gaze-aware applications.
Reference: Jindal, S., Yadav, M., & Manduchi, R. (2024). Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation. arXiv:2404.05215v2.
Summary updated