TL;DR
This paper introduces interpGaze, a novel framework for controllable gaze redirection that allows for both precise redirection to a target gaze and smooth, continuous interpolation between different gaze directions. By disentangling gaze attributes in a learned latent space, interpGaze offers fine-grained control over eye gaze manipulation, outperforming existing methods in image quality and redirection accuracy.
What Problem Does This Paper Address?
Eye contact is fundamental to social interaction, and manipulating gaze in images has numerous applications, from creating realistic group photos where everyone appears to be looking at the camera, to enhancing virtual communication. However, existing gaze redirection methods often struggle with generating photo-realistic results, especially for large redirection angles, and lack the ability to produce smooth, continuous transitions between different gazes. Some methods rely on complex 3D models requiring extensive data, while others fail to generalize to unseen angles or produce perceptually implausible results. This paper aims to bridge this gap by providing a unified framework for both precise gaze redirection and continuous gaze interpolation, addressing limitations in image quality, redirection precision, and the range of controllable gaze directions.
Methods
The interpGaze framework is built upon three core components: an encoder, a controller, and a decoder.
- Encoder: This component maps input eye images into a well-disentangled and hierarchically organized latent space. This disentanglement is crucial for separating different gaze attributes (head pose, yaw, pitch, and other miscellaneous attributes like glasses, skin color, etc.).
- Controller: This is the heart of the control mechanism. It takes the latent representations from the encoder and adjusts them based on a "control vector." This vector allows users to specify the desired strength of each attribute, enabling precise redirection by selecting target attributes or continuous interpolation by smoothly varying these strengths. The controller uses a novel morphing process that can be extended to a piece-wise formulation, allowing for flexible manipulation.
- Decoder: This component takes the manipulated latent features from the controller and reconstructs them back into a photo-realistic eye image.
The training process involves minimizing several loss functions, including adversarial loss (using WGAN-GP for stable training), perceptual loss (to ensure visual realism), reconstruction loss (for cycle consistency), and a new attribute classification loss to enforce the disentanglement of attributes in the latent space.
Datasets
The paper introduces a new, high-quality gaze image dataset for gaze redirection and related tasks. This dataset is designed to cover a larger range of eye angles than existing datasets and includes diversity in eye shapes, glasses, ages, and genders. The authors collected 29,250 images from 50 subjects in a controlled environment, using a grid of dots to precisely capture gaze directions. For training and testing, they utilized a split of their new dataset and also trained their model on the Columbia Gaze dataset for fair comparison.
Key Insights and Ideas
- Unified Framework for Redirection and Interpolation: The most significant insight is the unification of two seemingly distinct tasks (precise redirection and continuous interpolation) into a single, controllable framework. This is achieved by leveraging the disentangled latent space and the control vector.
- Attribute Disentanglement is Key: The success of
interpGazehinges on the ability to disentangle various gaze-related attributes (head pose, yaw, pitch) and other visual characteristics. This allows for independent control over each attribute, leading to more precise manipulation. - Control Vector for Fine-Grained Control: The introduction of a control vector provides an intuitive and powerful mechanism for users to specify the desired gaze attributes and their strengths, enabling both one-shot redirection and smooth interpolation.
- Importance of a Diverse, Large-Range Dataset: The authors highlight the necessity of a high-quality dataset with a wide range of gaze directions to train models that can generalize and cover the full gaze space, a limitation of previous datasets.
Key Results
- Superior Image Quality and Redirection Precision:
interpGazeoutperforms baseline methods like DeepWarp and He et al.'s GAN-based approach in terms of visual plausibility and redirection accuracy, as evidenced by lower LPIPS scores and user study results. - Continuous Interpolation and Extrapolation: The framework successfully generates smooth intermediate gaze images and can even extrapolate beyond the training data, demonstrating the effectiveness of the learned latent space.
- Preservation of Other Attributes: The method effectively preserves other visual attributes (e.g., eyebrows, glasses, skin color) during gaze manipulation, ensuring realistic and consistent results.
Limitations and Future Work
- Limitations: While the paper demonstrates impressive results, the authors acknowledge that the illustration of the control mechanism in Figure 4 is simplified. The precise control relies on the learned disentanglement and the ability to map latent space operations to attribute space operations. The "O" branch for miscellaneous attributes is mentioned but its full potential for multi-modal generation might require further exploration.
- Future Work: The authors suggest that their framework can be extended to integrate more attributes like illumination and exposure, as the current method shows consistency in preserving these aspects. The ability to perform extrapolation also opens doors for generating images with unseen target directions. The introduction of a new, comprehensive dataset is also a significant contribution that will benefit future research in this area.
Summary updated