Robust One-Shot Face Video Re-enactment using Hybrid Latent Spaces of StyleGAN2

Trevine Oorloff, Yaser YacoobICCV 2023 · DOI

Introduction

One-shot face re-enactment aims to animate a single source image using the motion from a driving video. Traditional methods often rely on explicit structural priors like 3D Morphable Models (3DMMs) or 2D landmarks. While these provide a baseline for motion, they frequently fail to capture fine-grained details—such as wrinkles, teeth, or eyeglasses—and can introduce spatio-temporal jitter.

Furthermore, many existing models lack one-shot robustness. Ideally, the output should depend solely on the source's identity. In practice, if a source photo contains a specific expression or pose, these attributes often "leak" into the generated video, causing unwanted distortions. This research introduces a high-fidelity (1024x1024) framework that achieves state-of-the-art robustness and seamless attribute editing without relying on explicit 3D models.

Methodology: Hybrid Latent Spaces and Cyclic Manifold Adjustment

The framework leverages the generative power of StyleGAN2 by decomposing facial representation into a "hybrid" latent space. This approach rests on three pillars:

  1. Hybrid Latent Space Decomposition: The model separates identity and motion into two distinct spaces. Identity is mapped to the $W+$ space for its balance of reconstruction and editability, while facial deformation is controlled via the first 10 layers of StyleSpace ($SS$). These layers are highly disentangled, capturing motion (eyes, mouth, pose) without altering the underlying identity.
  2. Self-Supervised Training: An encoder is trained to predict these latents. A "cross-identity" loss is employed during training to ensure that motion extracted from one individual can be applied to another without causing identity morphing.
  3. Cyclic Manifold Adjustment (CMA): To handle "out-of-domain" faces that a pre-trained StyleGAN2 might not perfectly represent, the authors introduce CMA. This inference-time optimization locally tweaks the generator's manifold, minimizing a cyclic reconstruction loss to ensure identity consistency and smooth motion transfer.

Key Results and Insights

The proposed method demonstrates significant improvements over existing baselines such as FOMM, LIA, and StyleHEAT.

  • High Resolution: The system operates at 1024x1024 resolution, a substantial leap from the 256x256 or 512x512 limits of many competitors.
  • Quantitative Superiority: The model outperforms state-of-the-art methods across metrics including L1 loss, LPIPS (perceptual quality), and FID (realism). Notably, it achieved up to a 50% improvement in one-shot robustness, effectively preventing source expression "leakage."
  • Simultaneous Editing: Because the model operates within StyleGAN2's latent spaces, users can apply standard GAN editing vectors (e.g., aging, adding a smile) concurrently with the re-enactment process.
Comparison of cross-identity re-enactment, attribute editing, and error maps against state-of-the-art methods.

Discussion and Limitations

While the framework sets a new standard for high-resolution re-enactment, it inherits certain StyleGAN2 characteristics. The most prominent is "texture sticking," where fine details like skin pores or beard textures appear fixed to the screen rather than moving naturally with the face. Additionally, like most GAN-based approaches, the model struggles with heavy occlusions (e.g., hands in front of the face) and complex backgrounds.

Future work may involve adapting this framework to StyleGAN3, which is designed to be more translation-invariant and could potentially mitigate the texture-sticking issue.

Summary updated

BibTeX
@inproceedings{oorloff2023robust,
  author = {Trevine Oorloff and Yaser Yacoob},
  title = {{Robust One-Shot Face Video Re-enactment using Hybrid Latent Spaces of StyleGAN2}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01915},
}
Figure