Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling

Abstract

This paper introduces SynthesEyes, a method for generating high-fidelity, photorealistic synthetic images of human eyes to train computer vision models. By using a sophisticated computer graphics pipeline—complete with refractive corneas, dynamic eyelids, and realistic lighting—the authors demonstrate that synthetic data can outperform models trained on manually labeled real-world data for tasks like eye-shape registration and gaze estimation.

Introduction

Training robust models for eye tracking and shape registration (detecting landmarks like eyelids and pupils) is notoriously difficult due to a "data bottleneck." Manual annotation is tedious and error-prone, 3D gaze vectors are nearly impossible for humans to label accurately from 2D images, and real-world data often lacks the environmental diversity needed for generalization. SynthesEyes addresses these challenges by bridging the "reality gap" with high-resolution 3D modeling and physically-based rendering.

Overview of the SynthesEyes pipeline, showing the dynamic model, rendered training images, and the resulting registration and estimation tasks.

Methodology

The authors developed a 3D eye-region model that moves beyond simple geometric approximations to capture the complexity of human anatomy:

  • Anatomical Eyeball: A two-part model featuring a refractive cornea and an inner iris/pupil to simulate realistic light interaction and reflections.
  • Dynamic Geometry: High-resolution (0.1mm) head scans were retopologized to allow for realistic skin movement. Blend shapes ensure eyelids naturally follow the eyeball's movement.
  • Image-Based Lighting (IBL): High Dynamic Range (HDR) panoramas of indoor and outdoor environments were used to simulate complex real-world lighting conditions.
  • Automated Labeling: Because the system is 3D, ground truth landmarks and gaze vectors are mathematically perfect and generated automatically during rendering.

Rendering was performed using Blender’s Cycles path-tracing engine, producing a dataset of 11,382 images.

Datasets

The primary contribution is the SynthesEyes dataset, which covers 90° of gaze variation and 40° of head pose variation across various eye colors and skin tones.

A grid of rendered training images demonstrating the diversity in gaze, lighting, and eye appearance provided by the SynthesEyes dataset.

To evaluate the synthetic data's utility, the authors tested their models on several real-world benchmarks:

  • 300-W: Used for evaluating eyelid registration "in the wild."
  • MPIIGaze: A large-scale dataset of laptop users, used for gaze estimation and iris tracking.
  • UT Multiview: A laboratory dataset used for cross-dataset comparison.

Results and Discussion

The study yielded several key insights into the use of synthetic data for computer vision:

  • Lighting is Crucial: Performance degrades significantly without realistic lighting. Models trained with diverse HDR environments generalize far better to real-world reflections.
  • Anatomical Coupling: Modeling the eyelid-gaze constraint was vital for the model to learn correct eye shapes at extreme gaze angles.
  • Synthetic-to-Real Transfer: A model trained entirely on synthetic data outperformed state-of-the-art methods on the 300-W dataset, likely because synthetic labels are perfectly consistent compared to noisy human annotations.
  • Performance Metrics: The CNN trained on SynthesEyes achieved a mean gaze error of 13.91° on MPIIGaze, comparable to models trained on real laboratory data. With minimal fine-tuning on real data, this error dropped to 7.90°, setting a new state-of-the-art for person-independent gaze estimation.

Conclusion

SynthesEyes demonstrates that high-quality synthetic data can effectively replace or augment real-world data for complex eye-tracking tasks. While the current model does not yet account for emotional expressions or complex occlusions (like glasses), it provides a scalable solution to the data bottleneck in eye-region analysis.

Summary updated

BibTeX
@inproceedings{wood2015rendering,
  author = {Erroll Wood and Tadas Baltrusaitis and Xucong Zhang and Yusuke Sugano and Peter Robinson and Andreas Bulling},
  title = {{Rendering of Eyes for Eye-Shape Registration and Gaze Estimation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=BybNqlfu-S},
}
Figure