Abstract
Gaze correction and animation in unconstrained "in the wild" images remain challenging due to the lack of precise annotations and the difficulty of maintaining identity consistency. We present GazeGAN, an unsupervised framework that treats gaze redirection as a dual in-painting task. By employing a Gaze Correction Module (GCM) and a Gaze Animation Module (GAM) alongside a novel Self-Supervised Mirror Learning technique, GazeGAN achieves state-of-the-art results in both realism and identity preservation without requiring manual head pose or gaze angle labels. Evaluation on the new CelebAGaze dataset demonstrates the model's superiority over existing supervised and unsupervised methods.
Introduction
Gaze correction—digitally adjusting eyes to look directly at a camera—is essential for natural videoconferencing and high-quality photo editing. However, performing this "in the wild" is difficult. Existing methods often require specialized hardware or massive, precisely labeled datasets with exact gaze and head angles. Furthermore, many models struggle with "identity consistency," where corrected eyes lose their original shape or color, or appear as blurry artifacts. GazeGAN addresses these issues using an unsupervised approach that works on standard high-resolution photos without needing angle labels.
Methodology
GazeGAN utilizes a tripartite architecture that treats gaze redirection as a dual in-painting problem, generating contextually appropriate pixels rather than warping existing ones.
Gaze Correction and Animation
The framework consists of two primary modules:
- Gaze Correction Module (GCM): Trained on images of subjects looking at the camera (Domain X), this module learns to mask out the eye region and "in-paint" forward-staring eyes.
- Gaze Animation Module (GAM): To enable arbitrary gaze directions, the authors introduced Synthesis-As-Training (SAT). The GCM creates paired data (original vs. corrected), which the GAM uses to learn a latent space where gaze direction can be controlled via vector interpolation.
Self-Supervised Mirror Learning
To ensure identity preservation, the Pretrained Autoencoder Module (PAM) employs Self-Supervised Mirror Learning. Recognizing that a person's left and right eyes share identity features (color, shape) despite different 2D gaze angles, the model extracts angle-invariant features by comparing an eye to the flipped version of the other. These features guide the in-painting modules to maintain the subject's identity.
CelebAGaze Dataset
The researchers introduced CelebAGaze, a benchmark derived from the CelebA dataset. It contains 25,283 high-resolution celebrity images, divided into Domain X (21,832 images staring at the camera) and Domain Y (3,451 images staring elsewhere). Unlike lab-captured datasets, CelebAGaze features diverse lighting, poses, and backgrounds, providing a robust "in the wild" testbed.
Results and Discussion
GazeGAN was evaluated against models such as StarGAN, CycleGAN, and PRGAN.
Qualitative Performance
GazeGAN produces significantly sharper results with fewer artifacts than its competitors. It successfully preserves iris color and eye shape, avoiding the "zombie-like" or mismatched eyes common in other models. As shown in the figure above, the model handles diverse subjects and head poses while maintaining high identity consistency.
Quantitative and User Evaluation
In terms of background preservation (measured by MSSSIM and LPIPS), GazeGAN achieved near-perfect scores (1.0 for MSSSIM). A user study with 30 respondents showed a strong preference for GazeGAN, which was chosen 42.4% of the time, significantly outperforming CycleGAN (15%) and PRGAN (8.33%).
Conclusion
GazeGAN demonstrates that treating gaze redirection as an in-painting task combined with mirror learning effectively solves the identity consistency problem in an unsupervised manner. While the model excels at static images, future work will focus on maintaining temporal consistency for video applications and handling extreme head angles through integration with 3D morphable models.
Summary updated