Abstract
Image editing using a pretrained StyleGAN generator has emerged as a powerful paradigm for facial editing, providing disentangled controls over age, expression, illumination, etc. However, the approach cannot be directly adopted for video manipulations. We hypothesize that the main missing ingredient is the lack of fine-grained and disentangled control over face location, face pose, and local facial expressions. In this work, we demonstrate that such a fine-grained control is indeed achievable using pretrained StyleGAN by working across multiple (latent) spaces (namely, the positional space, the W+ space, and the S space) and combining the optimization results across the multiple spaces. Building on this enabling component, we introduce Video2StyleGAN that takes a target image and driving video(s) to reenact the local and global locations and expressions from the driving video in the identity of the target image. We evaluate the effectiveness of our method over multiple challenging scenarios and demonstrate clear improvements over alternative approaches.
Introduction
While pretrained StyleGAN models excel at static face editing, extending these capabilities to video remains a significant challenge. The primary obstacle is achieving true disentanglement: modifying a subject's expression or head pose without inadvertently altering their core identity or introducing unnatural artifacts. Existing methods often suffer from either over-regularization, leading to stiff motion, or under-regularization, resulting in identity drift and "uncanny valley" deformations. Video2StyleGAN addresses these issues by introducing a hierarchical control framework that separates global movements from local facial expressions.
Methodology
The Video2StyleGAN framework leverages the architectural strengths of StyleGAN3, particularly its translation and rotation invariance. The pipeline employs a divide-and-conquer strategy across three distinct latent spaces:
- Positional Space (Canonical Transformation): Basic translation and rotation are handled via the Fourier features in StyleGAN3's initial layer. By utilizing landmark detection on the driving video, the system mathematically shifts and rotates the generated face to match the source without affecting facial features.
- W+ Space (Global Pose Encoding): Complex 3D-like pose changes (yaw and pitch) are managed by optimizing the first eight layers of the $W+$ latent space. A pose regression model ensures orientation alignment, while an identity preservation loss maintains the target subject's appearance.
- S Space (Local Facial Feature Encoding): Fine-grained expressions are controlled in the StyleSpace ($S$ space). A semantic segmentation network (BiSeNet) identifies specific channels corresponding to facial regions (eyes, mouth, nose), allowing for the precise transfer of local movements from a driving video to the target identity.
Results and Discussion
The effectiveness of Video2StyleGAN was evaluated using the MEAD (Multi-view Emotional Audio-visual Dataset) and compared against standard $W+$ space manipulation baselines.
Quantitative analysis demonstrates that Video2StyleGAN significantly outperforms baselines, achieving a lower FID (6.15 vs. 17.34) and improved Identity Distance (0.24 vs. 0.46). Qualitatively, the method successfully transfers complex expressions—such as shouting or squinting—while preserving the target identity and reducing artifacts common in prior work, such as shifting skin tones or distorted face shapes.
Conclusion
Video2StyleGAN demonstrates that fine-grained, disentangled control over facial video is achievable by strategically utilizing multiple latent spaces within StyleGAN3. While currently specialized for facial reenactment and requiring an initial optimization step, the framework provides a robust foundation for high-fidelity video editing. Future work will explore extending these techniques to full-body motion and integrating them with other generative architectures like Diffusion models.
Summary updated