TL;DR
ViTGaze is a lightweight, single-modality gaze-following framework that repurposes the self-attention maps of pre-trained Vision Transformers (ViTs) to model human-scene interactions. By treating attention as a direct proxy for "interaction," it achieves state-of-the-art performance while using 59% fewer parameters than complex multi-modal competitors.
What Problem Does This Paper Address?
Gaze following—predicting exactly where a person is looking in an image—is a challenging task that requires understanding the relationship between a person's head orientation and the objects in their environment.
Historically, researchers have taken two paths:
- Multi-modal approaches: These use extra information like depth maps or 3D head poses. While accurate, they require a "two-stage" design where the final prediction is only as good as the initial depth/pose estimation.
- Single-modal approaches: These use only RGB images but often rely on heavy, complex decoders to "learn" interactions from scratch, leading to high computational costs.
The authors of ViTGaze ask a clever question: Can we skip the complex decoders and extra modalities by leveraging the internal "knowledge" already present in a pre-trained Vision Transformer?
Methods
The core philosophy of ViTGaze is that self-attention is interaction. In a standard ViT, every image patch "attends" to every other patch. If a patch containing a person's eyes has a high attention score for a patch containing a coffee cup, the model has already implicitly "discovered" a potential gaze interaction.
The architecture consists of three main parts:
- 4D Interaction Encoder: Instead of just using the final feature map, ViTGaze extracts multi-level 4D attention maps (representing correlations between all pairs of patches) from a pre-trained ViT (like DINOv2).
- 2D Spatial Guidance Module: This module uses the provided head bounding box to generate spatial weights. It ensures the model focuses on the attention maps relevant to the specific person being analyzed.
- Lightweight Prediction Heads: Because the encoder does the heavy lifting of understanding interactions, the decoder is incredibly simple, accounting for less than 1% of the total parameters. It simply regresses a gaze heatmap and predicts whether the person is looking at something outside the frame.
Datasets
The model was trained and evaluated on the two most widely used benchmarks in the field:
- GazeFollow: A large-scale dataset containing over 130,000 annotations of people and their gaze targets in 122,000 images.
- VideoAttentionTarget: A dataset of 1,331 video clips with 165,000 annotations, used to test the model's ability to handle dynamic scenes and "out-of-frame" gaze targets.
Key Insights and Ideas
- Repurposing Attention: The most significant "aha moment" is the realization that the $L \times L$ attention matrix (where $L$ is the number of patches) is a 4D structure that inherently describes human-scene interactions. This allows the model to capture geometric relationships without needing explicit depth or pose data.
- The Power of Self-Supervision: The authors found that ViTs pre-trained with self-supervised methods (like DINOv2 or iBOT) are significantly better at gaze following than those trained with standard supervision (like ImageNet labels). Self-supervision seems to force the model to learn more robust, task-agnostic patch correlations.
- Encoder-Centric Design: Most computer vision tasks focus on designing better decoders. ViTGaze flips this, showing that if you extract the right features from a powerful encoder, the "decoding" part becomes trivial.
Key Results
ViTGaze sets a new bar for efficiency and accuracy:
- State-of-the-Art (SOTA): It achieved a 3.4% improvement in AUC (Area Under Curve) on GazeFollow and a 5.1% improvement in Average Precision (AP) on VideoAttentionTarget compared to previous single-modality methods.
- Efficiency: It uses only 22 million parameters. For context, it performs comparably to (or better than) multi-modal methods that use over 50 million parameters and require extra depth/pose inputs.
- Visual Proof: Qualitative results show that the 2D spatial guidance successfully "locks onto" the person's face, while the multi-level attention maps accurately highlight the objects they are watching.
Limitations and Future Work
Despite its success, ViTGaze struggles in highly occluded environments—for example, when a person's head is partially hidden or the gaze target is blocked by another object. The authors suggest that future versions could be improved by integrating "gaze dynamics" (how gaze moves over time in video) and global saliency maps to better predict where a person might be looking when the direct line of sight is obscured.
Summary updated