Learning to Detect Head Movement in Unconstrained Remote Gaze Estimation in the Wild

Zhecan Wang, Jian Zhao, Cheng Lu, Han Huang, Fan Yang, Lianji Li, Yandong Guo

Abstract

Unconstrained remote gaze estimation remains challenging mostly due to its vulnerability to the large variability in head-pose. Prior solutions struggle to maintain reliable accuracy in unconstrained remote gaze tracking. Among them, appearance-based solutions demonstrate tremendous potential in improving gaze accuracy. However, existing works still suffer from head movement and are not robust enough to handle real-world scenarios. Especially most of them study gaze estimation under controlled scenarios where the collected datasets often cover limited ranges of both head-pose and gaze which introduces further bias. In this paper, we propose novel end-to-end appearance-based gaze estimation methods that could more robustly incorporate different levels of head-pose representations into gaze estimation. Our method could generalize to real-world scenarios with low image quality, different lightings and scenarios where direct head-pose information is not available. To better demonstrate the advantage of our methods, we further propose a new benchmark dataset with the most rich distribution of head-gaze combination reflecting real-world scenarios. Extensive evaluations on several public datasets and our own dataset demonstrate that our method consistently outperforms the state-of-the-art by a significant margin.

TL;DR

This paper introduces a robust approach to remote gaze estimation that explicitly accounts for head movement, a major hurdle in real-world "unconstrained" settings. The authors propose two novel neural architectures—HGD (Head-pose aware Gaze Detector) and HGD-noHP—alongside a massive new In-car Gaze dataset to bridge the gap between controlled lab results and the messy reality of the wild.

The Challenge: Head Movement in the Wild

Gaze estimation—determining exactly where a person is looking—is a cornerstone of Human-Computer Interaction (HCI). While current systems work well when a user is sitting still in front of a monitor, they often fail in "unconstrained" environments, such as a moving vehicle.

The primary culprit is head movement. When a subject's head moves, the appearance of their eyes changes due to geometric deformation and occlusion, even if their gaze direction relative to the camera remains constant. Most existing datasets are collected in highly controlled environments with limited head-pose variety, leading to models that are biased and fragile. This research aims to make gaze estimation robust to these variations without requiring complex, multi-camera setups or personal calibration.

Proposed Methods

The authors propose two main frameworks based on the availability of head-pose information:

  1. HGD (Head-pose aware Gaze Detector): Utilized when both facial images and head-pose labels are available. Unlike previous methods that append a head-pose vector at the end of a network, HGD merges head and gaze information at multiple levels: input (face + eyes), hidden features (concatenation), and tasks (parallel learning). This ensures the model learns the deep relationship between head orientation and eye appearance.
  2. HGD-noHP: Designed for scenarios where direct head-pose data is missing. This model uses a ResNet-34 backbone to detect eye landmarks, extracting features from these "deformations" to implicitly infer head pose and gaze direction simultaneously.
The HGD architecture merges face and eye features to robustly estimate head pose and gaze angle.

A key technical innovation is BEC (Both Eyes Concatenated). The authors found that concatenating both eyes on the channel level (stacking them like layers of an image) outperformed horizontal or vertical stitching, providing the best balance of accuracy and computational efficiency.

A New Benchmark: The In-car Gaze Dataset

While the study utilizes standard benchmarks like Columbia Gaze and MPII Gaze, it highlights their limitations in head-pose diversity. To address this, the authors introduced the In-car Gaze Dataset.

The new In-car Gaze dataset captures diverse head-gaze combinations in real-world driving scenarios.

This new dataset features 1,000 participants, 400,000 frames, and 4 near-infrared cameras positioned at different angles within a vehicle. It provides the richest distribution of head-gaze combinations to date, specifically reflecting the challenging lighting and pose variations found in real-world driving scenarios.

Key Insights and Results

  • Representation Alignment: The authors argue that 3D head-pose vectors are mathematically distinct from learned image features. By merging them at "similar levels of representation" (hidden feature to hidden feature), the network learns the interaction more effectively than traditional late-stage fusion.
  • Eyes as a Proxy for Pose: HGD-noHP demonstrates that geometric deformation of the eye (iris shape, pupil position relative to lids) contains enough signal to "reverse-engineer" head orientation without needing a full face image.
  • The "Both Eyes" Advantage: Using both eyes allows the model to learn from asymmetry. Because the distance between pupils causes different gaze angles for each eye, channel-wise concatenation (BEC) makes the model more robust than single-eye alternatives.

Performance Metrics

The proposed methods consistently outperformed State-of-the-Art (SOTA) models:

  • Columbia Gaze: HGD achieved an Angle Error Metric (AEM) of 0.82°, a significant improvement over the previous SOTA (iTracker at 4.1°).
  • In-car Gaze: In the challenging vehicle environment, HGD (using BEC) achieved an AEM of 1.79°, while the standard iTracker model struggled with an error of 5.61°.
  • Efficiency: BEC was not only more accurate but also more computationally efficient (0.32 GFLOPS) compared to other binocular merging techniques (~0.62 GFLOPS).

Conclusion

By explicitly modeling the relationship between head pose and eye appearance, the HGD framework significantly advances the state of remote gaze estimation. The introduction of the In-car Gaze dataset provides a vital resource for future research into unconstrained gaze tracking, moving the field closer to reliable deployment in complex, real-world environments.

Summary updated

Abstract

Unconstrained remote gaze estimation remains challenging mostly due to its vulnerability to the large variability in head-pose. Prior solutions struggle to maintain reliable accuracy in unconstrained remote gaze tracking. Among them, appearance-based solutions demonstrate tremendous potential in improving gaze accuracy. However, existing works still suffer from head movement and are not robust enough to handle real-world scenarios. Especially most of them study gaze estimation under controlled scenarios where the collected datasets often cover limited ranges of both head-pose and gaze which introduces further bias. In this paper, we propose novel end-to-end appearance-based gaze estimation methods that could more robustly incorporate different levels of head-pose representations into gaze estimation. Our method could generalize to real-world scenarios with low image quality, different lightings and scenarios where direct head-pose information is not available. To better demonstrate the advantage of our methods, we further propose a new benchmark dataset with the most rich distribution of head-gaze combination reflecting real-world scenarios. Extensive evaluations on several public datasets and our own dataset demonstrate that our method consistently outperforms the state-of-the-art by a significant margin.
BibTeX
@inproceedings{wang2020learning,
  author = {Zhecan Wang and Jian Zhao and Cheng Lu and Han Huang and Fan Yang and Lianji Li and Yandong Guo},
  title = {{Learning to Detect Head Movement in Unconstrained Remote Gaze Estimation in the Wild}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=P_mbrcMmUU},
}
Figure