TransGOP: Transformer-Based Gaze Object Prediction

Binglu Wang, Chenxi Guo, Yang Jin, Haisheng Xia, Nian LiuAAAI 2024 · DOI

TL;DR

This paper introduces TransGOP, a novel end-to-end framework that leverages Transformers for Gaze Object Prediction (GOP). By integrating a Transformer-based object detector with a Transformer-based gaze regressor, TransGOP achieves state-of-the-art performance in accurately predicting not only where a person is looking but also the specific object they are attending to, which is crucial for applications in retail and human-computer interaction.

What Problem Does This Paper Address?

Predicting where a person is looking, known as Gaze Estimation (GE), has long been a focus in computer vision. However, GE typically only outputs a heatmap or point indicating the general gaze direction, failing to identify the specific object being viewed. This limitation hinders applications that require understanding what a person is interested in, such as retail analytics (identifying products of interest) or assistive technologies for individuals with communication impairments.

The Gaze Object Prediction (GOP) task was introduced to address this by predicting both the location and category of the gazed-at object. Existing GOP methods, like GaTector, primarily rely on Convolutional Neural Networks (CNNs) for object detection. While effective, CNNs can struggle with dense object scenes common in retail environments. Furthermore, the relationship between the human observer and the object of their gaze is often a long-range dependency, which Transformers, with their inherent ability to model global context, are better suited to capture. This paper aims to bridge this gap by proposing a Transformer-centric approach for GOP, enhancing both object detection accuracy in challenging scenarios and the prediction of gaze targets.

Methods

TransGOP is an end-to-end framework composed of two main components: an Object Detector and a Gaze Regressor.

  1. Transformer-based Object Detector: The paper utilizes an off-the-shelf Transformer-based object detector, specifically DINO (Zhang et al., 2022), which is known for its strong performance and faster convergence compared to earlier DETR variants. This detector takes the entire scene image as input and outputs bounding boxes and categories for all detected objects. The loss for this component, $L_{det}$, includes classification and box regression losses.
  2. Transformer-based Gaze Regressor: This is the core novelty of TransGOP. It takes the head image, scene image, and a head location map as input.
  3. Gaze Feature Fusion: Similar to previous work, it fuses features from the head and scene images using a ResNet50 backbone and a gaze fusion module.
  4. Gaze Autoencoder: A Transformer-based gaze autoencoder is employed to model long-range dependencies within gaze-related features. This autoencoder uses self-attention mechanisms to process fused gaze features.
  5. Object-to-Gaze Cross-Attention: To improve gaze heatmap regression, a novel object-to-gaze cross-attention mechanism is introduced. This mechanism allows the gaze autoencoder's queries to leverage "global-memory position knowledge" from the object detector's encoder outputs (keys and values). This helps the gaze regressor better understand the spatial context of detected objects.
  6. Gaze Predictor: A regression head predicts the final gaze heatmap. The loss for this component is $L_{gaze}$.
  7. Gaze Box Loss ($L_{gb}$): To enable end-to-end training and jointly optimize both the object detector and the gaze regressor, a new Gaze Box loss is proposed. This loss focuses on maximizing the heatmap energy within the ground truth bounding box of the gazed-at object. This is crucial because the gradient backpropagation from the gaze heatmap can influence the object detector through the gaze fusion module, allowing for joint optimization.

The overall training objective is a weighted sum of these losses: $L = L_{det} + \alpha L_{gaze} + \beta L_{gb}$.

Datasets

The experiments were conducted on two datasets:

  • GOO-Synth: A large dataset of synthetic images of people gazing at objects in retail environments.
  • GOO-Real: A smaller dataset of real-world images of people in retail settings.

These datasets were originally introduced by Tomas et al. (2021) for the GOP task.

Key Insights and Ideas

  • Transformers Excel in Dense Object Scenarios: The paper empirically demonstrates that Transformer-based object detectors outperform CNN-based ones in dense retail scenes, likely due to their superior long-range modeling capabilities. This insight motivates their use in the GOP task.
  • Long-Range Dependencies are Key for GOP: The Transformer's ability to capture long-range relationships between the human head and the gaze object is crucial for improving prediction accuracy. The gaze autoencoder is designed to exploit this.
  • Bridging Object Detection and Gaze Regression: The object-to-gaze cross-attention mechanism is a key innovation that directly links the outputs of the object detector to the gaze regressor, allowing the gaze model to learn from the precise object localization information.
  • End-to-End Joint Optimization: The Gaze Box loss is critical for making the entire TransGOP framework trainable end-to-end. It ensures that improvements in gaze prediction also positively impact object detection, and vice-versa.

Key Results

  • State-of-the-Art GOP Performance: TransGOP achieves state-of-the-art results on both GOO-Synth and GOO-Real datasets across various metrics (e.g., mSoC), significantly outperforming the previous best method, GaTector. For instance, on GOO-Synth, TransGOP achieves 92.8% mSoC compared to GaTector's 67.9%.
  • Improved Object Detection: While using an off-the-shelf detector (DINO), TransGOP's object detection performance also improves compared to DINO alone, demonstrating the benefit of joint optimization via the Gaze Box loss.
  • Enhanced Gaze Estimation: TransGOP also shows strong performance in gaze estimation metrics (AUC, L2 distance, angle error), often surpassing previous methods, especially when pre-training is not used.
  • Qualitative Improvements: Visualizations show TransGOP's ability to detect objects more accurately in cluttered scenes and to predict more precise gaze heatmaps that align better with the detected object boxes.

Limitations and Future Work

  • Dataset Dependency: While TransGOP performs well on GOO-Synth and GOO-Real, its generalization to significantly different environments (e.g., non-retail settings) is not extensively discussed.
  • Computational Cost: Transformer-based models, while powerful, can be computationally intensive. The paper uses a single GPU for training, suggesting potential for further optimization.
  • Failure Cases: The paper acknowledges failure cases in visualization, indicating that challenges remain in highly complex or ambiguous scenarios.

Future work could involve exploring more efficient Transformer architectures, extending the framework to diverse environments, and investigating more sophisticated gaze-object interaction modeling.

Summary updated

BibTeX
@inproceedings{wang2024transgop,
  author = {Binglu Wang and Chenxi Guo and Yang Jin and Haisheng Xia and Nian Liu},
  title = {{TransGOP: Transformer-Based Gaze Object Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28883},
}
Figure