Subject Guided Eye Image Synthesis With Application to Gaze Redirection

Harsimran Kaur, Roberto ManduchiWACV 2021

TL;DR

This paper introduces a novel method for synthesizing realistic eye images with a desired style and gaze direction, using a "style image" and a segmentation mask. This approach enables gaze redirection without requiring manually annotated real-world gaze data, achieving state-of-the-art results by leveraging synthetic masks and a cyclic training mechanism for style consistency.

What Problem Does This Paper Address?

Synthesizing realistic human eye images with specific gaze directions is crucial for applications like videoconferencing, photo editing, and generating training data for gaze estimation algorithms. Traditional computer graphics methods often produce unnatural-looking images. While recent works have focused on synthesizing realistic eye images, they often lack fine-grained control over specific attributes like skin color, texture, and iris color, which are essential for maintaining the identity and appearance of the person in the image. Furthermore, accurately annotating gaze direction in real-world images is time-consuming and error-prone. This paper addresses the challenge of generating realistic eye images with controllable style and gaze direction, and importantly, enabling gaze redirection without relying on expensive, manually annotated real-world gaze data.

Methods

The core of the proposed method is a Style-Based Eye Image Synthesis module. This module takes an input "style image" (which defines the desired appearance attributes like skin color, texture, iris color, etc.) and a ternary segmentation mask (representing the desired gaze direction) to generate a new eye image. The generator, based on the SPADE architecture, is trained using a cyclic mechanism to ensure both style consistency with the input style image and content consistency (desired gaze direction).

To achieve gaze redirection without manual gaze annotations, the authors introduce a Gaze Redirection via Mask Synthesis approach. This involves a mask redirection network that takes a segmentation mask from an input image and a desired gaze direction variation, producing a new, redirected mask. This synthesized mask is then fed into the Style-Based Eye Image Synthesis module to generate the final gaze-redirected eye image. This bypasses the need for real-world gaze annotations by relying on synthetically generated masks.

Key components of the training process include:

  • Synthesis Loss: A perceptual loss using VGG-16 features to ensure the generated image is similar to a target image.
  • Re-synthesis Loss: A cycle-consistency loss that feeds the synthesized image back into the generator with the original mask to reconstruct the original image, enforcing style preservation.
  • Adversarial Loss: A standard GAN loss using a discriminator to improve image realism.
  • Feature Matching Loss: An additional discriminator-based loss to further improve the quality of generated images.
  • Mask Synthesis Loss: A cross-entropy loss for training the mask redirection network to produce accurate redirected masks.
  • Mask Re-synthesis Loss: Another cross-entropy loss to ensure the mask redirection network can reconstruct the original mask when given a reversed gaze direction change.

Datasets

  • Columbia Gaze Dataset: Used for training and testing the Style-Based Eye Image Synthesis module. This dataset contains facial images of 56 subjects with recorded gaze data under constrained settings.
  • UnityEyes: Used to generate synthetic eye images and masks. These synthetic masks are crucial for training the mask redirection network and for providing the input gaze direction masks for the synthesis module. The paper notes that while UnityEyes provides synthetic masks, they may not be perfectly suited to the "style" of real images, motivating their mask synthesis approach.

Key Insights and Ideas

  • Style as a Controllable Attribute: The paper frames eye image synthesis as a problem of controlling both content (gaze direction) and style (appearance attributes), moving beyond generic style transfer.
  • Masks as Style-Independent Proxies for Gaze: Ternary segmentation masks are effectively used to represent gaze direction, decoupling it from specific visual features and allowing for manipulation.
  • Synthetic Masks for Gaze Redirection: The most significant insight is the ability to perform gaze redirection without requiring manually annotated real-world gaze data. By training a network to synthesize accurate gaze-redirected masks from input masks, the system can then leverage the style-based synthesis module to produce realistic images.
  • Cyclic Training for Style Preservation: The re-synthesis loss is a clever way to enforce that the generated image retains the style of the input image, a critical requirement for realistic gaze redirection.

Key Results

  • Style-Based Eye Image Synthesis: The proposed method achieves the lowest FID score (a measure of image quality and diversity) compared to existing methods like EyeGAN, CycleGAN, and SimGAN. It also achieves a competitive mIoU score, indicating good semantic consistency.
  • Gaze Redirection: The method achieves state-of-the-art results in gaze redirection, outperforming a previous method (dubbed "GazeRedirGAN") in terms of LPIPS metric, especially when trained on a reduced dataset. Qualitative results demonstrate visually convincing gaze redirection while preserving the original eye's style.
  • No Real-World Gaze Annotations Needed: A key achievement is demonstrating effective gaze redirection using only synthetic gaze labels for training, significantly reducing data acquisition costs.

Limitations and Future Work

  • Mask Quality Dependence: The performance of the gaze redirection heavily relies on the quality of the synthesized masks. While the mask redirection network is trained to be robust, imperfections in the initial segmentation or the redirection process could impact the final image.
  • "Style" Definition: While the paper defines "style" broadly as appearance attributes, a more precise quantification or disentanglement of these attributes could lead to even finer control.
  • Future Work: The authors suggest that their approach could be extended to more complex scenarios, such as full face synthesis with redirected gaze, or to handle more challenging real-world conditions like varying illumination and occlusions. The robustness of the mask redirection network to diverse input conditions is also an area for further exploration.

Summary updated

BibTeX
@inproceedings{kaur2021subject,
  author = {Harsimran Kaur and Roberto Manduchi},
  title = {{Subject Guided Eye Image Synthesis With Application to Gaze Redirection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure