Integrating Human Gaze into Attention for Egocentric Activity Recognition

Kyle Min, Jason J. Corso

Introduction

Egocentric video, captured from head-mounted cameras, provides a unique perspective on human activity. A key cue in these videos is the wearer's gaze, which indicates the focus of attention. However, integrating raw gaze data into deep learning models is challenging due to measurement noise (e.g., blinks, saccades), the dissociation between physical gaze and mental attention, and the lack of eye-tracking hardware in real-world scenarios. This paper proposes a probabilistic framework that treats gaze as a latent variable, allowing the model to leverage gaze information during training while remaining functional without it during inference.

Methods: Probabilistic Gaze Modeling

The proposed architecture is built upon a two-stream I3D backbone that processes both RGB and optical flow frames. The core innovation lies in modeling gaze as a structured discrete latent variable.

The proposed architecture integrates a two-stream I3D backbone with a variational gaze modeling component. Gaze is represented as a latent variable z, which is sampled to generate a spatiotemporal attention map.

Variational Gaze Integration

Instead of simple coordinate regression, the model represents gaze as a sequence of 2D discrete random variables. During training, the model is supervised by ground-truth gaze data to learn a variational distribution. At test time, the model predicts this distribution directly from video features. To ensure stable training with discrete variables, the authors employ a direct optimization technique for unbiased gradient estimation, avoiding the biases often introduced by methods like Gumbel-Softmax.

Soft Attention Mechanism

The sampled latent gaze points are transformed into a spatiotemporal "soft attention" map via a fully-connected layer and sigmoid activation. This map is then used to re-weight intermediate convolutional features, guiding the network to focus on task-relevant regions.

Key Insights and Results

The model was evaluated on the EGTEA and GTEA Gaze+ datasets, achieving state-of-the-art performance with 62.84% and 64.81% mean class accuracy, respectively.

Qualitative comparison of attention maps. 'Ours' demonstrates more focused and task-relevant attention compared to the baseline I3D and even models forced to use ground-truth gaze (I3D w/ Gaze), which can be distracted by noisy eye movements.

The Gaze-Attention Paradox

A significant finding is that the model often performs better using its predicted gaze distribution than using actual ground-truth gaze at test time. This occurs because the latent model learns to filter out "noisy" gaze points—such as looking at a background object—and focuses on patterns that are truly predictive of the activity. This robustness to dissociation allows the internal attention mechanism to remain "locked" on relevant objects like tools or ingredients even when the user's eyes wander.

Conclusion

By treating gaze as a latent spatiotemporal sequence, this framework effectively bridges the gap between raw eye-tracking data and high-level activity recognition. While the computational cost of 3D latent modeling and the requirement for gaze-annotated training data are current limitations, the approach sets a new benchmark for egocentric video understanding.

Summary updated

BibTeX
@inproceedings{min2021integrating,
  author = {Kyle Min and Jason J. Corso},
  title = {{Integrating Human Gaze into Attention for Egocentric Activity Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=ug_rYHbzNGI},
}
Figure