Abstract
Given a face image, most previous works in gaze estimation infer the gaze via a well-trained model with supervised training. However, the distribution of test data may be very different compared to that of training data since samples might be corrupted in real-world scenarios (e.g., taking a photo in strong light). This leads to a gap between the source domain (training data) and target domain (test data). In this paper, we introduce self-supervised learning into our method for addressing challenging situations in gaze estimation. Moreover, existing appearance-based gaze estimation methods focus on developing powerful regressors using face and eye images simultaneously or face/eye images only, often overlooking the inter-cues between face and eye features. To this end, we propose a novel Modulation-based Adaptive Network (MANet) for gaze estimation, which uses high-level knowledge to filter distractive information and bridges the intrinsic relationship between face and eye features. Further, we combine self-supervised learning and MANet to learn to adapt to challenging cases, such as abnormal lighting conditions and poor-quality images, by minimizing a self-supervised loss and a supervised loss jointly. The experimental results on several datasets demonstrate the effectiveness of our proposed approach with a real-time speed of 900 fps on a PC with an NVIDIA Titan RTX GPU.
Introduction
Gaze estimation—predicting where a person is looking based on a face image—is vital for applications in VR, human-computer interaction, and saliency detection. However, most models struggle with domain shift. A model trained on high-quality, well-lit lab data often fails when faced with "in-the-wild" images corrupted by Gaussian noise, motion blur, or extreme lighting.
Furthermore, while humans use both the eyes and the overall face to determine gaze, most AI models simply concatenate these features. This "simple addition" ignores the intrinsic relationship between the two: eye regions provide precise cues, while the face provides rich but often "noisy" context (like head pose).
Methodology
The proposed solution consists of two primary components:
1. Modulation-based Adaptive Network (MANet)
Instead of simple concatenation, MANet utilizes an Eye-Guiding Network to extract high-level cues from eye patches. These cues are transformed into "modulator vectors" that perform channel-wise multiplication on the face features. This mechanism uses the eyes to "filter" out distractive information from the face image, focusing the model on the most relevant visual data.
2. Auxiliary Self-Learning
Inspired by the BYOL (Bootstrap Your Own Latent) framework, an auxiliary self-supervised task is added during training. The model employs an "online" network and a "target" network, forcing the system to produce similar feature representations for an original image and its corrupted or augmented version. This constraint ensures the model's feature space remains stable even when input quality drops. The final loss function is a joint optimization of the standard supervised gaze error (angular loss) and the self-supervised consistency loss.
Experimental Results
The model was evaluated on four major benchmark datasets: GazeCapture, MPIIGaze, EyeDiap, and Gaze360.
- State-of-the-Art Performance: In cross-dataset evaluation (training on GazeCapture and testing on MPIIGaze), the method achieved an angular error of 4.37°, outperforming previous benchmarks like ETH-XGaze by over 3%.
- Robustness to Corruption: Ablation studies confirmed that the combination of the eye-guiding network and self-learning significantly reduces error in poor-quality images. For instance, on subject P00, the error dropped from 3.56° to 3.00° with these features enabled.
- Efficiency: The inference model is highly efficient, achieving a real-time speed of 900 fps on an NVIDIA Titan RTX, making it practical for real-world deployment.
Conclusion and Future Work
The MANet framework demonstrates that treating eye features as a gate for face features, combined with auxiliary self-supervision, effectively bridges the gap between lab-trained models and real-world data. Future work will focus on automating eye region localization to create a fully end-to-end pipeline and exploring more diverse data augmentation techniques to further enhance robustness.
Summary updated