Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation

Yunyang Xiong, Hyunwoo J. Kim, Vikas SinghCVPR 2019 · OpenReview

Abstract

There is much interest in computer vision to utilize commodity hardware for gaze estimation. A number of papers have shown that algorithms based on deep convolutional architectures are approaching accuracies where streaming data from mass-market devices can offer good gaze tracking performance, although a gap still remains between what is possible and the performance users will expect in real deployments. We observe that one obvious avenue for improvement relates to a gap between some basic technical assumptions behind most existing approaches and the statistical properties of the data used for training. Specifically, most training datasets involve tens of users with a few hundreds (or more) repeated acquisitions per user. The non i.i.d. nature of this data suggests better estimation may be possible if the model explicitly made use of such "repeated measurements" from each user as is commonly done in classical statistical analysis using so-called mixed effects models. The goal of this paper is to adapt these "mixed effects" ideas from statistics within a deep neural network architecture for gaze estimation, based on eye images. Such a formulation seeks to specifically utilize information regarding the hierarchical structure of the training data -- each node in the hierarchy is a user who provides tens or hundreds of repeated samples. This modification yields an architecture that offers state of the art performance on various publicly available datasets improving results by 10-20%.

Introduction: The i.i.d. Assumption in Gaze Estimation

Most deep learning models are built on the fundamental assumption that data samples are independent and identically distributed (i.i.d.). However, in fields like gaze estimation, this assumption is almost always violated. A typical dataset consists of thousands of images, but they are often collected from only a few dozen participants. Because each person has unique physical characteristics—such as eye shape, bone structure, or head posture—the relationship between an eye image and a gaze direction is not universal.

Standard models attempt to force a single "global" solution for everyone, which leads to a significant performance gap. This paper argues that gaze datasets should be treated as hierarchical: multiple "repeated measurements" nested within individual subjects. Failing to account for these individual variations introduces noise into the global model.

Conceptual illustration of mixed effects: individual subjects (Alice, Bob, etc.) exhibit unique baseline offsets and trends, violating the i.i.d. assumption.

Methods: Mixed Effects Neural Networks (MeNets)

The authors introduce Mixed Effects Neural Networks (MeNets) to bridge the gap between classical statistics and deep learning. The core innovation is the decomposition of the prediction into two distinct components:

  1. Fixed Effects: These are the "global" parameters shared across the entire population, capturing the general relationship between eye appearance and gaze.
  2. Random Effects: These are subject-specific adjustments or "offsets" that account for individual variations. For example, one user's eyes might always appear slightly more "downward" than another's due to eyelid shape.

Architecture and Training

The MeNet architecture utilizes a standard backbone (such as ResNet-18) to extract a hidden feature representation. This feature is then fed into a mixed-effects regression layer that separates global and subject-specific components.

The MeNet architecture: a deep backbone extracts features which are then processed by fixed effect (FC(β)) and subject-specific random effect (FC(u_i)) layers.

Training a MeNet requires a more sophisticated approach than standard backpropagation. The authors employ a Variational Expectation-Maximization (EM) algorithm combined with Stochastic Gradient Descent (SGD). In the E-step, the model estimates subject-specific random effects, while the M-step updates the global network weights and variance parameters. To handle new users at test time, the model learns a proxy function to predict the likely "random effect offset" based solely on the input image, allowing for calibration-free personalization.

Key Results and Performance

MeNets were evaluated on several prominent benchmarks, including MPIIGaze and UT Multiview. The results demonstrate that even if random effects are not explicitly used at test time, training with them makes the global "fixed" effects much more accurate by accounting for individual noise during the learning process.

  • MPIIGaze: The model achieved an error of 4.90°, a significant improvement over the standard ResNet baseline of 6.04°.
  • UT Multiview: MeNets improved results by 10-20% across various settings.
  • Within-Subject Accuracy: In scenarios where the model had seen the subject during training, the error dropped by as much as 30% (2.66° vs 3.9°).

Conclusion

MeNets provide a blueprint for handling hierarchical data structures in deep learning. By incorporating statistical rigor into neural architectures, the model achieves state-of-the-art performance in gaze estimation. Future work will focus on combining MeNets with formal domain adaptation techniques to further reduce the gap between synthetic training data and real-world applications, as well as integrating temporal dynamics for video-based tracking.

Summary updated

Abstract

There is much interest in computer vision to utilize commodity hardware for gaze estimation. A number of papers have shown that algorithms based on deep convolutional architectures are approaching accuracies where streaming data from mass-market devices can offer good gaze tracking performance, although a gap still remains between what is possible and the performance users will expect in real deployments. We observe that one obvious avenue for improvement relates to a gap between some basic technical assumptions behind most existing approaches and the statistical properties of the data used for training. Specifically, most training datasets involve tens of users with a few hundreds (or more) repeated acquisitions per user. The non i.i.d. nature of this data suggests better estimation may be possible if the model explicitly made use of such "repeated measurements" from each user as is commonly done in classical statistical analysis using so-called mixed effects models. The goal of this paper is to adapt these "mixed effects" ideas from statistics within a deep neural network architecture for gaze estimation, based on eye images. Such a formulation seeks to specifically utilize information regarding the hierarchical structure of the training data -- each node in the hierarchy is a user who provides tens or hundreds of repeated samples. This modification yields an architecture that offers state of the art performance on various publicly available datasets improving results by 10-20%.
BibTeX
@inproceedings{xiong2019mixed,
  author = {Yunyang Xiong and Hyunwoo J. Kim and Vikas Singh},
  title = {{Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=te5OyAsa1eV},
}
Figure