Abstract
Unconstrained remote gaze estimation remains challenging mostly due to its vulnerability to the large variability in head-pose. Prior solutions struggle to maintain reliable accuracy in unconstrained remote gaze tracking. Among them, appearance-based solutions demonstrate tremendous potential in improving gaze accuracy. However, existing works still suffer from head movement and are not robust enough to handle real-world scenarios. Especially most of them study gaze estimation under controlled scenarios where the collected datasets often cover limited ranges of both head-pose and gaze which introduces further bias. In this paper, we propose novel end-to-end appearance-based gaze estimation methods that could more robustly incorporate different levels of head-pose representations into gaze estimation. Our method could generalize to real-world scenarios with low image quality, different lightings and scenarios where direct head-pose information is not available. To better demonstrate the advantage of our methods, we further propose a new benchmark dataset with the most rich distribution of head-gaze combination reflecting real-world scenarios. Extensive evaluations on several public datasets and our own dataset demonstrate that our method consistently outperforms the state-of-the-art by a significant margin.
Introduction
Remote gaze estimation—determining exactly where a person is looking using only a camera—is a task that transitions from simple in a laboratory setting to highly complex in real-world environments. The primary challenge is unconstrained head movement. Gaze is a composite vector derived from the eye's orientation relative to the head and the head's orientation relative to the camera.
Existing appearance-based models often fail due to appearance confusion (where similar eye appearances map to different targets due to head tilt), geometric distortion of the eye region, and dataset biases from controlled collection environments. This paper introduces the Head-pose aware Gaze Detector (HGD) to address these issues by deeply integrating head-pose data into the neural network architecture.
Methodology
The researchers propose a multi-level fusion approach to incorporate head-pose information more effectively than previous state-of-the-art methods.
HGD (Head-pose aware Gaze Detector)
HGD is an end-to-end model that merges head and gaze information at four distinct levels: input (face and eyes), hidden features (latent vector concatenation), model (parallel backbones), and task (joint prediction of pose and gaze). By using a ResNet-34 backbone, the framework achieves high accuracy with a low computational footprint of 0.32 GFLOPS, making it suitable for real-time automotive safety applications.
Both Eyes Concatenated (BEC)
A significant finding of this work is the benefit of processing both eyes simultaneously. Instead of the traditional single-eye method, HGD concatenates both eyes into a single input channel. This allows the model to learn the inherent asymmetry and correlation between the eyes, providing an approximate 1-degree boost in accuracy by resolving ambiguities caused by head tilt.
HGD-noHP
For scenarios where full-face images or explicit head-pose labels are unavailable, the authors developed HGD-noHP. This variant extracts head-pose information implicitly from the geometric deformation of the eye region. It utilizes a landmark detector to identify 16 interior eye and iris points, which serve as effective proxies for head orientation.
Datasets and Benchmarking
To validate the model in "wild" conditions, the authors introduced the In-car Gaze Dataset. This massive benchmark includes 400,000 frames from 1,000 participants, captured using near-infrared cameras inside vehicles. Unlike standard datasets like MPIIGaze or GazeCapture, this dataset covers a much wider range of head-gaze combinations and includes challenging real-world factors such as nighttime lighting, sunglasses, and hats.
Key Results
Extensive evaluations demonstrate that HGD consistently outperforms existing methods:
- Public Benchmarks: On the Columbia Gaze dataset, HGD achieved an Angle Error Metric (AEM) of 0.82°, a significant improvement over the 5.42° baseline.
- In-Car Performance: On the new, more challenging dataset, HGD achieved an AEM of 1.79°, compared to 5.61° for the popular iTracker framework.
Conclusion
The HGD framework demonstrates that robust gaze estimation requires more than just "tacking on" head-pose data at the final layer. By merging representations at similar levels of complexity and leveraging the relationship between both eyes, HGD achieves superior performance in unconstrained environments. The release of the In-car Gaze dataset provides a vital resource for future research into high-movement, real-world gaze tracking.
Summary updated