Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance

Jingjing Chen, Jichao Zhang, Enver Sangineto, Tao Chen, Jiayuan Fan, Nicu Sebe

Abstract

Gaze redirection is a critical task in computer vision with applications ranging from video conferencing to photography. However, existing methods often struggle to balance image realism with redirection precision. This paper presents CFGR, a novel coarse-to-fine framework that combines spatial warping with adversarial residual learning. By integrating both numerical angles and pictorial gazemaps as guidance, CFGR achieves state-of-the-art performance in both perceptual quality and gaze accuracy.

Introduction

Digital gaze redirection involves altering the eye direction of a subject in a photograph while maintaining their identity and the natural appearance of the eye region. This task is inherently difficult because it requires large-scale spatial transformations (moving the iris) alongside the preservation of fine-scale textures (eyelids, skin, and reflections).

Previous approaches typically rely on either complex 3D modeling, which often results in "uncanny" renderings, or standard Generative Adversarial Networks (GANs), which frequently produce blurry artifacts or fail to meet target gaze angles precisely. The authors identify that single-stage models are often overwhelmed by the dual requirements of transformation and synthesis. CFGR addresses this by decoupling the process into two specialized stages.

Methodology

The CFGR framework utilizes a Coarse-to-Fine strategy to handle the complexities of gaze redirection through two main branches:

  1. The Coarse Branch (Warping): This stage focuses on the global movement of the eye. Using an encoder-decoder architecture, the model learns a "flow field" to warp existing pixels from the source image to the target position. This approach is highly effective at preserving the subject's identity and basic eye structure.
  2. The Fine Branch (Residual Learning): Because warping can introduce artifacts or blurriness, a second branch utilizes a conditional GAN to learn a residual image. This residual acts as a correction layer, added to the warped image to restore high-frequency textures and fix distortions.
The CFGR architecture overview, illustrating the coarse warping branch (Enc/Dec) and the fine residual learning branch (Generator G) guided by numerical and pictorial inputs.

Numerical and Pictorial Guidance (NPG)

A key innovation of this work is the use of dual guidance. While most models rely solely on numerical angles (e.g., "look 15 degrees left"), CFGR also provides a pictorial gazemap. This 2D mask serves as a spatial blueprint, indicating the desired position of the eyeball and iris, which simplifies the learning task for the neural network and significantly improves redirection precision.

Multi-task Discriminator

To ensure both realism and accuracy, the framework employs a multi-task discriminator. Beyond distinguishing real images from generated ones, it performs gaze regression to verify that the generated eye is pointing in the correct target direction.

Experimental Results

The model was evaluated using the Columbia Gaze Dataset, consisting of 5,880 high-resolution images. CFGR was compared against two prominent baselines: DeepWarp and PRGAN.

Quantitative Performance

CFGR outperformed existing methods across both quality and accuracy metrics:

  • Perceptual Quality (LPIPS): CFGR achieved a score of 0.0333, notably better than PRGAN (0.0409) and DeepWarp (0.0946).
  • Gaze Precision: The redirection error was reduced to 5.15°, compared to 5.37° for PRGAN and a much higher 14.18° for DeepWarp.

Qualitative Analysis

Visual comparisons demonstrate that CFGR produces sharper images with fewer artifacts than its predecessors. In a user study, participants preferred CFGR results 52.78% of the time, citing superior realism and accuracy.

Qualitative comparison of gaze redirection results across different methods. CFGR demonstrates superior texture preservation and redirection accuracy compared to DeepWarp and PRGAN.

Conclusion

By combining spatial warping with residual synthesis and introducing pictorial guidance, the CFGR framework effectively bridges the gap between identity preservation and gaze accuracy. While the current model excels in controlled environments, future work will focus on extending this robustness to "in the wild" scenarios involving unconstrained lighting and extreme head poses.

Summary updated

Abstract

Gaze redirection aims at manipulating the gaze of a given face image with respect to a desired direction (i.e., a reference angle) and it can be applied to many real life scenarios, such as video-conferencing or taking group photos. However, previous work on this topic mainly suffers of two limitations: (1) Low-quality image generation and (2) Low redirection precision. In this paper, we propose to alleviate these problems by means of a novel gaze redirection framework which exploits both a numerical and a pictorial direction guidance, jointly with a coarse-to-fine learning strategy. Specifically, the coarse branch learns the spatial transformation which warps input image according to desired gaze. On the other hand, the fine-grained branch consists of a generator network with conditional residual image learning and a multi-task discriminator. This second branch reduces the gap between the previously warped image and the ground-truth image and recovers finer texture details. Moreover, we propose a numerical and pictorial guidance module (NPG) which uses a pictorial gazemap description and numerical angles as an extra guide to further improve the precision of gaze redirection. Extensive experiments on a benchmark dataset show that the proposed method outperforms the state-of-the-art approaches in terms of both image quality and redirection precision. The code is available at https://github.com/jingjingchen777/CFGR
BibTeX
@inproceedings{chen2021coarse,
  author = {Jingjing Chen and Jichao Zhang and Enver Sangineto and Tao Chen and Jiayuan Fan and Nicu Sebe},
  title = {{Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00371},
}
Figure