Abstract
This paper introduces SynthesEyes, a method for generating high-fidelity, photorealistic synthetic images of human eyes to train computer vision models. By using a sophisticated computer graphics pipeline—complete with refractive corneas, dynamic eyelids, and realistic lighting—the authors demonstrate that synthetic data can outperform models trained on manually labeled real-world data for tasks like eye-shape registration and gaze estimation.
Introduction
Training robust models for eye tracking and shape registration (detecting landmarks like eyelids and pupils) is notoriously difficult due to a "data bottleneck." Manual annotation is tedious and error-prone, 3D gaze vectors are nearly impossible for humans to label accurately from 2D images, and real-world data often lacks the environmental diversity needed for generalization. SynthesEyes addresses these challenges by bridging the "reality gap" with high-resolution 3D modeling and physically-based rendering.
Methodology
The authors developed a 3D eye-region model that moves beyond simple geometric approximations to capture the complexity of human anatomy:
- Anatomical Eyeball: A two-part model featuring a refractive cornea and an inner iris/pupil to simulate realistic light interaction and reflections.
- Dynamic Geometry: High-resolution (0.1mm) head scans were retopologized to allow for realistic skin movement. Blend shapes ensure eyelids naturally follow the eyeball's movement.
- Image-Based Lighting (IBL): High Dynamic Range (HDR) panoramas of indoor and outdoor environments were used to simulate complex real-world lighting conditions.
- Automated Labeling: Because the system is 3D, ground truth landmarks and gaze vectors are mathematically perfect and generated automatically during rendering.
Rendering was performed using Blender’s Cycles path-tracing engine, producing a dataset of 11,382 images.
Datasets
The primary contribution is the SynthesEyes dataset, which covers 90° of gaze variation and 40° of head pose variation across various eye colors and skin tones.
To evaluate the synthetic data's utility, the authors tested their models on several real-world benchmarks:
- 300-W: Used for evaluating eyelid registration "in the wild."
- MPIIGaze: A large-scale dataset of laptop users, used for gaze estimation and iris tracking.
- UT Multiview: A laboratory dataset used for cross-dataset comparison.
Results and Discussion
The study yielded several key insights into the use of synthetic data for computer vision:
- Lighting is Crucial: Performance degrades significantly without realistic lighting. Models trained with diverse HDR environments generalize far better to real-world reflections.
- Anatomical Coupling: Modeling the eyelid-gaze constraint was vital for the model to learn correct eye shapes at extreme gaze angles.
- Synthetic-to-Real Transfer: A model trained entirely on synthetic data outperformed state-of-the-art methods on the 300-W dataset, likely because synthetic labels are perfectly consistent compared to noisy human annotations.
- Performance Metrics: The CNN trained on SynthesEyes achieved a mean gaze error of 13.91° on MPIIGaze, comparable to models trained on real laboratory data. With minimal fine-tuning on real data, this error dropped to 7.90°, setting a new state-of-the-art for person-independent gaze estimation.
Conclusion
SynthesEyes demonstrates that high-quality synthetic data can effectively replace or augment real-world data for complex eye-tracking tasks. While the current model does not yet account for emotional expressions or complex occlusions (like glasses), it provides a scalable solution to the data bottleneck in eye-region analysis.
Summary updated