ViTGaze: Gaze Following with Interaction Features in Vision Transformers

Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, Xiangmin Xu

TL;DR

ViTGaze is a lightweight, single-modality gaze-following framework that repurposes the self-attention maps of pre-trained Vision Transformers (ViTs) to model human-scene interactions. By treating attention as a direct proxy for "interaction," it achieves state-of-the-art performance while using 59% fewer parameters than complex multi-modal competitors.

What Problem Does This Paper Address?

Gaze following—predicting exactly where a person is looking in an image—is a challenging task that requires understanding the relationship between a person's head orientation and the objects in their environment.

Historically, researchers have taken two paths:

  1. Multi-modal approaches: These use extra information like depth maps or 3D head poses. While accurate, they require a "two-stage" design where the final prediction is only as good as the initial depth/pose estimation.
  2. Single-modal approaches: These use only RGB images but often rely on heavy, complex decoders to "learn" interactions from scratch, leading to high computational costs.

The authors of ViTGaze ask a clever question: Can we skip the complex decoders and extra modalities by leveraging the internal "knowledge" already present in a pre-trained Vision Transformer?

Methods

The core philosophy of ViTGaze is that self-attention is interaction. In a standard ViT, every image patch "attends" to every other patch. If a patch containing a person's eyes has a high attention score for a patch containing a coffee cup, the model has already implicitly "discovered" a potential gaze interaction.

The architecture consists of three main parts:

  • 4D Interaction Encoder: Instead of just using the final feature map, ViTGaze extracts multi-level 4D attention maps (representing correlations between all pairs of patches) from a pre-trained ViT (like DINOv2).
  • 2D Spatial Guidance Module: This module uses the provided head bounding box to generate spatial weights. It ensures the model focuses on the attention maps relevant to the specific person being analyzed.
  • Lightweight Prediction Heads: Because the encoder does the heavy lifting of understanding interactions, the decoder is incredibly simple, accounting for less than 1% of the total parameters. It simply regresses a gaze heatmap and predicts whether the person is looking at something outside the frame.
The overall architecture of ViTGaze, showing how 4D interaction features are extracted and guided by head position.

Datasets

The model was trained and evaluated on the two most widely used benchmarks in the field:

  • GazeFollow: A large-scale dataset containing over 130,000 annotations of people and their gaze targets in 122,000 images.
  • VideoAttentionTarget: A dataset of 1,331 video clips with 165,000 annotations, used to test the model's ability to handle dynamic scenes and "out-of-frame" gaze targets.

Key Insights and Ideas

  • Repurposing Attention: The most significant "aha moment" is the realization that the $L \times L$ attention matrix (where $L$ is the number of patches) is a 4D structure that inherently describes human-scene interactions. This allows the model to capture geometric relationships without needing explicit depth or pose data.
  • The Power of Self-Supervision: The authors found that ViTs pre-trained with self-supervised methods (like DINOv2 or iBOT) are significantly better at gaze following than those trained with standard supervision (like ImageNet labels). Self-supervision seems to force the model to learn more robust, task-agnostic patch correlations.
  • Encoder-Centric Design: Most computer vision tasks focus on designing better decoders. ViTGaze flips this, showing that if you extract the right features from a powerful encoder, the "decoding" part becomes trivial.
Comparison of how ViTGaze uses attention maps versus how previous tasks used standard feature maps.

Key Results

ViTGaze sets a new bar for efficiency and accuracy:

  • State-of-the-Art (SOTA): It achieved a 3.4% improvement in AUC (Area Under Curve) on GazeFollow and a 5.1% improvement in Average Precision (AP) on VideoAttentionTarget compared to previous single-modality methods.
  • Efficiency: It uses only 22 million parameters. For context, it performs comparably to (or better than) multi-modal methods that use over 50 million parameters and require extra depth/pose inputs.
  • Visual Proof: Qualitative results show that the 2D spatial guidance successfully "locks onto" the person's face, while the multi-level attention maps accurately highlight the objects they are watching.
A comparison plot showing ViTGaze (top right) achieving the highest AUC scores with a much smaller parameter count (indicated by circle size).

Limitations and Future Work

Despite its success, ViTGaze struggles in highly occluded environments—for example, when a person's head is partially hidden or the gaze target is blocked by another object. The authors suggest that future versions could be improved by integrating "gaze dynamics" (how gaze moves over time in video) and global saliency maps to better predict where a person might be looking when the direct line of sight is obscured.

Summary updated

Abstract

Gaze following aims to interpret human-scene interactions by predicting the person's focal point of gaze. Prevailing approaches often adopt a two-stage framework, whereby multi-modality information is extracted in the initial stage for gaze target prediction. Consequently, the efficacy of these methods highly depends on the precision of the preceding modality extraction. Others use a single-modality approach with complex decoders, increasing network computational load. Inspired by the remarkable success of pre-trained plain vision transformers (ViTs), we introduce a novel single-modality gaze following framework called ViTGaze. In contrast to previous methods, it creates a novel gaze following framework based mainly on powerful encoders (relative decoder parameters less than 1%). Our principal insight is that the inter-token interactions within self-attention can be transferred to interactions between humans and scenes. Leveraging this presumption, we formulate a framework consisting of a 4D interaction encoder and a 2D spatial guidance module to extract human-scene interaction information from self-attention maps. Furthermore, our investigation reveals that ViT with self-supervised pre-training has an enhanced ability to extract correlation information. Many experiments have been conducted to demonstrate the performance of the proposed method. Our method achieves state-of-the-art (SOTA) performance among all single-modality methods (3.4% improvement in the area under curve (AUC) score, 5.1% improvement in the average precision (AP)) and very comparable performance against multi-modality methods with 59% number of parameters less.
BibTeX
@inproceedings{song2024vitgaze,
  author = {Yuehao Song and Xinggang Wang and Jingfeng Yao and Wenyu Liu and Jinglin Zhang and Xiangmin Xu},
  title = {{ViTGaze: Gaze Following with Interaction Features in Vision Transformers}},
  booktitle = {Vis. Intell},
  year = {2024},
}
Figure