Learning a Generalized Gaze Estimator from Gaze-Consistent Feature

Mingjie Xu, Haofei Wang, Feng LuAAAI 2023 · DOI

Introduction

Gaze estimation—determining where a person is looking based on a face image—is a critical task in computer vision with applications ranging from human-computer interaction to driver monitoring. While modern deep learning models perform exceptionally well when test data matches the training distribution, they often struggle to generalize to new individuals, facial expressions, or lighting conditions. This paper, presented at AAAI-23, addresses this domain generalization challenge by isolating gaze-consistent features from environmental noise.

The Domain Generalization Challenge

Current gaze estimation models are highly sensitive to "noise" factors such as identity, facial expression, and illumination. Because the visual signal for gaze (eyeball orientation) is subtle compared to these dominant facial features, models often overfit to gaze-irrelevant factors. For instance, a model trained in a controlled laboratory setting may fail in real-world environments due to differences in lighting or ethnic diversity.

Overview of the domain generalization problem in gaze estimation, showing the error increase when moving from within-domain to cross-domain testing.

The goal is to train a model on a source domain ($D_s$) such that it maintains high accuracy on an unseen target domain ($D_t$). As shown in the figure above, cross-domain estimation error typically increases significantly compared to within-domain error due to these domain gaps.

Methodology: Extracting Gaze-Consistent Features

The authors propose a framework consisting of two primary phases to decouple gaze information from irrelevant facial attributes.

1. Gaze-irrelevant Feature Disturbation (GiFD)

Rather than relying solely on existing data diversity, the researchers actively "disturb" training samples to isolate four key irrelevant factors: Identity, Expression, Illumination, and Tone.

  • Identity, Expression, and Illumination: These are scrambled using adversarial attacks (specifically the Fast Gradient Sign Method, or FGSM). The attack targets the model's auxiliary branches for these factors while keeping the gaze label constant, effectively creating a version of the image where the gaze is preserved but other features are noisy.
  • Tone: Standard data augmentation techniques like ColorJitter are used to vary the image's color profile.

2. Gaze-consistent Feature Extraction (GcFE)

The core of the approach involves a ResNet-18 backbone that processes both the original and the "disturbed" images. To ensure the model focuses only on gaze, the authors employ Maximum Mean Discrepancy (MMD) to minimize the distance between the feature representations of the original and disturbed pairs. By forcing these features to be identical, the model is compelled to discard any information that changed during disturbance (like identity) and retain only the consistent signal: the gaze.

Experimental Results

The framework was evaluated across four major gaze datasets: ETH-XGaze, Gaze360, MPIIGaze, and EyeDiap.

  • State-of-the-Art Performance: The method achieved significant improvements in cross-domain scenarios. For example, training on ETH-XGaze and testing on MPIIGaze resulted in an angular error of 6.50°, outperforming previous benchmarks like PureGaze (8.80°).
  • Feature Explainability: t-SNE visualizations confirmed that the learned features cluster primarily by gaze direction, indicating that the model successfully ignored identity-driven variations.
  • Efficient Fine-tuning: In supervised domain adaptation tasks, the gaze-consistent pre-training allowed for faster convergence and lower final error rates when limited target data was available.

Conclusion

By treating adversarial attacks as a "feature purifier," this work demonstrates a robust method for improving the generalization of gaze estimators. While the current model focuses on four primary irrelevant factors, the resulting architecture offers a high-performance, deployable solution for real-world gaze tracking that is less sensitive to the diverse reality of human faces and environments.

Summary updated

BibTeX
@inproceedings{xu2023learning,
  author = {Mingjie Xu and Haofei Wang and Feng Lu},
  title = {{Learning a Generalized Gaze Estimator from Gaze-Consistent Feature}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25406},
}
Figure