Abstract
Gaze redirection is a critical task in computer vision with applications ranging from video conferencing to photography. However, existing methods often struggle to balance image realism with redirection precision. This paper presents CFGR, a novel coarse-to-fine framework that combines spatial warping with adversarial residual learning. By integrating both numerical angles and pictorial gazemaps as guidance, CFGR achieves state-of-the-art performance in both perceptual quality and gaze accuracy.
Introduction
Digital gaze redirection involves altering the eye direction of a subject in a photograph while maintaining their identity and the natural appearance of the eye region. This task is inherently difficult because it requires large-scale spatial transformations (moving the iris) alongside the preservation of fine-scale textures (eyelids, skin, and reflections).
Previous approaches typically rely on either complex 3D modeling, which often results in "uncanny" renderings, or standard Generative Adversarial Networks (GANs), which frequently produce blurry artifacts or fail to meet target gaze angles precisely. The authors identify that single-stage models are often overwhelmed by the dual requirements of transformation and synthesis. CFGR addresses this by decoupling the process into two specialized stages.
Methodology
The CFGR framework utilizes a Coarse-to-Fine strategy to handle the complexities of gaze redirection through two main branches:
- The Coarse Branch (Warping): This stage focuses on the global movement of the eye. Using an encoder-decoder architecture, the model learns a "flow field" to warp existing pixels from the source image to the target position. This approach is highly effective at preserving the subject's identity and basic eye structure.
- The Fine Branch (Residual Learning): Because warping can introduce artifacts or blurriness, a second branch utilizes a conditional GAN to learn a residual image. This residual acts as a correction layer, added to the warped image to restore high-frequency textures and fix distortions.
Numerical and Pictorial Guidance (NPG)
A key innovation of this work is the use of dual guidance. While most models rely solely on numerical angles (e.g., "look 15 degrees left"), CFGR also provides a pictorial gazemap. This 2D mask serves as a spatial blueprint, indicating the desired position of the eyeball and iris, which simplifies the learning task for the neural network and significantly improves redirection precision.
Multi-task Discriminator
To ensure both realism and accuracy, the framework employs a multi-task discriminator. Beyond distinguishing real images from generated ones, it performs gaze regression to verify that the generated eye is pointing in the correct target direction.
Experimental Results
The model was evaluated using the Columbia Gaze Dataset, consisting of 5,880 high-resolution images. CFGR was compared against two prominent baselines: DeepWarp and PRGAN.
Quantitative Performance
CFGR outperformed existing methods across both quality and accuracy metrics:
- Perceptual Quality (LPIPS): CFGR achieved a score of 0.0333, notably better than PRGAN (0.0409) and DeepWarp (0.0946).
- Gaze Precision: The redirection error was reduced to 5.15°, compared to 5.37° for PRGAN and a much higher 14.18° for DeepWarp.
Qualitative Analysis
Visual comparisons demonstrate that CFGR produces sharper images with fewer artifacts than its predecessors. In a user study, participants preferred CFGR results 52.78% of the time, citing superior realism and accuracy.
Conclusion
By combining spatial warping with residual synthesis and introducing pictorial guidance, the CFGR framework effectively bridges the gap between identity preservation and gaze accuracy. While the current model excels in controlled environments, future work will focus on extending this robustness to "in the wild" scenarios involving unconstrained lighting and extreme head poses.
Summary updated