PCFGaze: Physics-Consistent Feature for Appearance-based Gaze Estimation

Yiwei Bao, Feng LuarXiv 2023 · DOI · arXiv

Introduction

Gaze estimation is a fundamental task for applications ranging from human-computer interaction to medical diagnostics. While appearance-based methods using deep learning offer flexibility with standard webcams, they often struggle with a "generalization gap" when moving from controlled training environments to diverse real-world settings. Traditional solutions rely on domain adaptation, requiring target-domain data. In contrast, PCFGaze focuses on domain generalization, leveraging the inherent physical geometry of eye movement to create a robust, "out-of-the-box" solution.

Overview of the PCFGaze framework showing the transition from traditional black-box regression to a physics-consistent approach.

Methodology: The PCFGaze Framework

The core innovation of PCFGaze is the alignment of the neural network's feature space with the physical world. The authors discovered that the "gaze feature manifold" in a trained network mirrors the spherical nature of the eye. Specifically, the geodesic distance between features is proportional to the actual angular difference in gaze.

Physics-Consistent Feature (PCF) and Spherical Fitting

To exploit this, the framework introduces Physics-Consistent Features (PCF). Using manifold learning techniques like Isomap, high-dimensional gaze features are reduced to a 3D spherical shell. Instead of a standard numerical regressor, PCFGaze employs Spherical Fitting (SF)—an analytical method that projects features onto a sphere to calculate pitch and yaw using geometric principles.

PCF-Oriented Training

To make this process efficient and differentiable for training, the authors developed an Isometric Propagator. This small multi-layer perceptron (MLP) simulates the Isomap transformation, allowing a ResNet-18 backbone to be trained to produce features that naturally adhere to the physical spherical manifold.

Detailed breakdown of the PCFGaze motivation, spherical fitting process, and the PCF-oriented training pipeline.

Experimental Results

The framework was evaluated across four major benchmarks: ETH-XGaze ($D_E$), Gaze360 ($D_G$), MPIIFaceGaze ($D_M$), and EyeDiap ($D_D$). PCFGaze consistently outperformed state-of-the-art generalization methods across multiple cross-dataset scenarios.

  • Accuracy Improvements: The model achieved up to a 29.23% error reduction in the transfer from MPIIFaceGaze to EyeDiap compared to baselines.
  • Zero-Shot Generalization: Performance gains were achieved without any exposure to target-domain data during training, demonstrating the power of physical constraints.
  • Training Stability: By replacing black-box regression with analytical projection, the model exhibited lower variance and higher stability across training epochs.

Conclusion

PCFGaze demonstrates that incorporating physical constraints into deep learning architectures can significantly enhance robustness and interpretability. By forcing features to follow the geometry of the physical task, the model avoids overfitting to domain-specific artifacts like lighting or camera resolution. Future work may extend this "physics-consistent" paradigm to other regression tasks governed by known physical laws, such as human pose estimation or object orientation tracking.

Summary updated

Abstract

Although recent deep learning based gaze estimation approaches have achieved much improvement, we still know little about how gaze features are connected to the physics of gaze. In this paper, we try to answer this question by analyzing the gaze feature manifold. Our analysis revealed the insight that the geodesic distance between gaze features is consistent with the gaze differences between samples. According to this finding, we construct the Physics- Consistent Feature (PCF) in an analytical way, which connects gaze feature to the physical definition of gaze. We further propose the PCFGaze framework that directly optimizes gaze feature space by the guidance of PCF. Experimental results demonstrate that the proposed framework alleviates the overfitting problem and significantly improves cross-domain gaze estimation accuracy without extra training data. The insight of gaze feature has the potential to benefit other regression tasks with physical meanings.
BibTeX
@misc{bao2023pcfgaze,
  author = {Yiwei Bao and Feng Lu},
  title = {{PCFGaze: Physics-Consistent Feature for Appearance-based Gaze Estimation}},
  year = {2023},
  eprint = {2309.02165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.02165},
}
Figure