TL;DR
This paper introduces EyeGAN, a novel generative adversarial network (GAN) for synthesizing realistic eye images with controllable gaze direction. By leveraging ternary masks as domain-independent proxies for gaze, EyeGAN overcomes limitations of previous methods that struggled with photometric differences and direct image comparisons, leading to more accurate gaze preservation.
What Problem Does This Paper Address?
Generating realistic eye images with a specific gaze direction is crucial for applications like teleconferencing and, importantly, for creating large, annotated datasets to train gaze tracking algorithms. Manually annotating gaze direction in real images is challenging and error-prone. Existing methods, like SimGAN, attempt domain transfer from synthetic to real images but suffer from photometric inconsistencies that can bias gaze direction estimation. The core problem is how to synthesize realistic eye images that accurately preserve a desired gaze direction without relying on difficult-to-obtain annotations from the target domain.
Methods
EyeGAN employs a conditional GAN architecture, inspired by pix2pix, to translate ternary masks into realistic eye images. The key innovation is the use of ternary masks (representing iris, sclera, and surrounding skin) as the input to the generator, rather than a full synthetic image. These masks are domain-independent proxies for gaze and head orientation.
The training process involves:
- Generator: Takes a ternary mask as input and outputs a synthetic eye image in the target domain's style.
- Discriminator: Distinguishes between real target domain images and generated images.
- Loss Function: A combination of adversarial loss (for realism) and an L1 loss term that penalizes the difference between the generated image and the real image from the target domain associated with the input mask. This L1 loss is computed within the target domain, avoiding cross-domain photometric biases.
- Segmenter Training: A crucial aspect is training a segmenter to extract ternary masks from real images. This segmenter is trained iteratively, first on synthetic data and then augmented with generated images from EyeGAN, leading to improved mask quality.
Datasets
- Synthetic Data: Generated using the UnityEyes tool, providing pairs of synthetic eye images and their corresponding ternary masks.
- Target Domains:
- UBIRIS: Used for gaze direction validation and eye region segmentation. Contains manual annotations of the iris region.
- SBVPI: Used for eye region segmentation. Contains iris, pupil, sclera, and periocular masks for all subjects.
- BioID: Used for pupil localization. Contains grayscale images with pupil center annotations.
- NVGaze: Used for gaze estimation. Contains both real and synthetic eye images with gaze direction annotations.
Key Insights and Ideas
- Ternary Masks as Gaze Proxies: The paper demonstrates that well-designed ternary masks contain sufficient information to guide the generation of realistic eye images with accurate gaze direction, acting as a robust, domain-independent representation.
- Within-Domain L1 Loss: By computing the L1 loss between generated images and their corresponding real target domain images (rather than comparing across domains), EyeGAN avoids photometric biases that plague methods like SimGAN, leading to better gaze consistency.
- Iterative Segmenter Training: The iterative approach to training the segmentation network, where generated images are used to augment training data, significantly improves the quality of masks extracted from real images.
- Masks for Training Data Generation: EyeGAN can generate realistic eye images with inherited annotations, which are shown to be superior for training downstream tasks like segmentation, pupil localization, and gaze estimation compared to using purely synthetic data or data generated by other domain transfer methods.
Key Results
- Gaze Consistency: EyeGAN achieved a gaze consistency index of 0.89, significantly outperforming SimGAN (0.48) and CycleGAN (0.36) in preserving the intended gaze direction.
- Eye Region Segmentation: Training segmentation models with EyeGAN-generated data resulted in performance comparable to or better than training on real target domain data (TT), particularly for the UBIRIS dataset. For example, mean IoU for skin segmentation was 0.94 with EyeGAN vs. 0.97 for TT.
- Pupil Localization: Training a pupil localization network with EyeGAN-generated data yielded the best results, outperforming baseline synthetic training and other domain transfer methods.
- Gaze Estimation: Training a gaze estimation network using EyeGAN-generated data with inherited annotations resulted in the lowest mean gaze angular error (5.3°) compared to baseline synthetic training (23°).
Limitations and Future Work
- Segmenter Quality Dependency: The performance of EyeGAN is dependent on the quality of the ternary masks. While the iterative training improves this, initial mask quality can still influence results.
- Computational Cost: Training GANs and iterative segmenter refinement can be computationally intensive.
- Future Work: The authors suggest exploring more sophisticated mask representations and investigating the use of EyeGAN for generating other facial attributes or for more complex domain transfer tasks. The potential for creating even larger and more diverse annotated datasets for various eye-related tasks is highlighted.
Summary updated