Controllable Continuous Gaze Redirection

Weihao Xia, Yujiu Yang, Jing-Hao Xue, Wensen Feng

TL;DR

This paper introduces interpGaze, a novel framework for controllable gaze redirection that allows for both precise redirection to a target gaze and smooth, continuous interpolation between different gaze directions. By disentangling gaze attributes in a learned latent space, interpGaze offers fine-grained control over eye gaze manipulation, outperforming existing methods in image quality and redirection accuracy.

What Problem Does This Paper Address?

Eye contact is fundamental to social interaction, and manipulating gaze in images has numerous applications, from creating realistic group photos where everyone appears to be looking at the camera, to enhancing virtual communication. However, existing gaze redirection methods often struggle with generating photo-realistic results, especially for large redirection angles, and lack the ability to produce smooth, continuous transitions between different gazes. Some methods rely on complex 3D models requiring extensive data, while others fail to generalize to unseen angles or produce perceptually implausible results. This paper aims to bridge this gap by providing a unified framework for both precise gaze redirection and continuous gaze interpolation, addressing limitations in image quality, redirection precision, and the range of controllable gaze directions.

Methods

The interpGaze framework is built upon three core components: an encoder, a controller, and a decoder.

  • Encoder: This component maps input eye images into a well-disentangled and hierarchically organized latent space. This disentanglement is crucial for separating different gaze attributes (head pose, yaw, pitch, and other miscellaneous attributes like glasses, skin color, etc.).
  • Controller: This is the heart of the control mechanism. It takes the latent representations from the encoder and adjusts them based on a "control vector." This vector allows users to specify the desired strength of each attribute, enabling precise redirection by selecting target attributes or continuous interpolation by smoothly varying these strengths. The controller uses a novel morphing process that can be extended to a piece-wise formulation, allowing for flexible manipulation.
  • Decoder: This component takes the manipulated latent features from the controller and reconstructs them back into a photo-realistic eye image.

The training process involves minimizing several loss functions, including adversarial loss (using WGAN-GP for stable training), perceptual loss (to ensure visual realism), reconstruction loss (for cycle consistency), and a new attribute classification loss to enforce the disentanglement of attributes in the latent space.

Datasets

The paper introduces a new, high-quality gaze image dataset for gaze redirection and related tasks. This dataset is designed to cover a larger range of eye angles than existing datasets and includes diversity in eye shapes, glasses, ages, and genders. The authors collected 29,250 images from 50 subjects in a controlled environment, using a grid of dots to precisely capture gaze directions. For training and testing, they utilized a split of their new dataset and also trained their model on the Columbia Gaze dataset for fair comparison.

Key Insights and Ideas

  • Unified Framework for Redirection and Interpolation: The most significant insight is the unification of two seemingly distinct tasks (precise redirection and continuous interpolation) into a single, controllable framework. This is achieved by leveraging the disentangled latent space and the control vector.
  • Attribute Disentanglement is Key: The success of interpGaze hinges on the ability to disentangle various gaze-related attributes (head pose, yaw, pitch) and other visual characteristics. This allows for independent control over each attribute, leading to more precise manipulation.
  • Control Vector for Fine-Grained Control: The introduction of a control vector provides an intuitive and powerful mechanism for users to specify the desired gaze attributes and their strengths, enabling both one-shot redirection and smooth interpolation.
  • Importance of a Diverse, Large-Range Dataset: The authors highlight the necessity of a high-quality dataset with a wide range of gaze directions to train models that can generalize and cover the full gaze space, a limitation of previous datasets.

Key Results

  • Superior Image Quality and Redirection Precision: interpGaze outperforms baseline methods like DeepWarp and He et al.'s GAN-based approach in terms of visual plausibility and redirection accuracy, as evidenced by lower LPIPS scores and user study results.
  • Continuous Interpolation and Extrapolation: The framework successfully generates smooth intermediate gaze images and can even extrapolate beyond the training data, demonstrating the effectiveness of the learned latent space.
  • Preservation of Other Attributes: The method effectively preserves other visual attributes (e.g., eyebrows, glasses, skin color) during gaze manipulation, ensuring realistic and consistent results.

Limitations and Future Work

  • Limitations: While the paper demonstrates impressive results, the authors acknowledge that the illustration of the control mechanism in Figure 4 is simplified. The precise control relies on the learned disentanglement and the ability to map latent space operations to attribute space operations. The "O" branch for miscellaneous attributes is mentioned but its full potential for multi-modal generation might require further exploration.
  • Future Work: The authors suggest that their framework can be extended to integrate more attributes like illumination and exposure, as the current method shows consistency in preserving these aspects. The ability to perform extrapolation also opens doors for generating images with unseen target directions. The introduction of a new, comprehensive dataset is also a significant contribution that will benefit future research in this area.

Summary updated

Abstract

In this work, we present interpGaze, a novel framework for controllable gaze redirection that achieves both precise redirection and continuous interpolation. Given two gaze images with different attributes, our goal is to redirect the eye gaze of one person into any gaze direction depicted in the reference image or to generate continuous intermediate results. To accomplish this, we design a model including three cooperative components: an encoder, a controller and a decoder. The encoder maps images into a well-disentangled and hierarchically-organized latent space. The controller adjusts the magnitudes of latent vectors to the desired strength of corresponding attributes by altering a control vector. The decoder converts the desired representations from the attribute space to the image space. To facilitate covering the full space of gaze directions, we introduce a high-quality gaze image dataset with a large range of directions, which also benefits researchers in related areas. Extensive experimental validation and comparisons to several baseline methods show that the proposed interpGaze outperforms state-of-the-art methods in terms of image quality and redirection precision.
BibTeX
@inproceedings{xia2020controllable,
  author = {Weihao Xia and Yujiu Yang and Jing-Hao Xue and Wensen Feng},
  title = {{Controllable Continuous Gaze Redirection}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=FJw2mDcFVJD},
}
Figure