Introduction
One-shot face re-enactment aims to animate a single source image using the motion from a driving video. Traditional methods often rely on explicit structural priors like 3D Morphable Models (3DMMs) or 2D landmarks. While these provide a baseline for motion, they frequently fail to capture fine-grained details—such as wrinkles, teeth, or eyeglasses—and can introduce spatio-temporal jitter.
Furthermore, many existing models lack one-shot robustness. Ideally, the output should depend solely on the source's identity. In practice, if a source photo contains a specific expression or pose, these attributes often "leak" into the generated video, causing unwanted distortions. This research introduces a high-fidelity (1024x1024) framework that achieves state-of-the-art robustness and seamless attribute editing without relying on explicit 3D models.
Methodology: Hybrid Latent Spaces and Cyclic Manifold Adjustment
The framework leverages the generative power of StyleGAN2 by decomposing facial representation into a "hybrid" latent space. This approach rests on three pillars:
- Hybrid Latent Space Decomposition: The model separates identity and motion into two distinct spaces. Identity is mapped to the $W+$ space for its balance of reconstruction and editability, while facial deformation is controlled via the first 10 layers of StyleSpace ($SS$). These layers are highly disentangled, capturing motion (eyes, mouth, pose) without altering the underlying identity.
- Self-Supervised Training: An encoder is trained to predict these latents. A "cross-identity" loss is employed during training to ensure that motion extracted from one individual can be applied to another without causing identity morphing.
- Cyclic Manifold Adjustment (CMA): To handle "out-of-domain" faces that a pre-trained StyleGAN2 might not perfectly represent, the authors introduce CMA. This inference-time optimization locally tweaks the generator's manifold, minimizing a cyclic reconstruction loss to ensure identity consistency and smooth motion transfer.
Key Results and Insights
The proposed method demonstrates significant improvements over existing baselines such as FOMM, LIA, and StyleHEAT.
- High Resolution: The system operates at 1024x1024 resolution, a substantial leap from the 256x256 or 512x512 limits of many competitors.
- Quantitative Superiority: The model outperforms state-of-the-art methods across metrics including L1 loss, LPIPS (perceptual quality), and FID (realism). Notably, it achieved up to a 50% improvement in one-shot robustness, effectively preventing source expression "leakage."
- Simultaneous Editing: Because the model operates within StyleGAN2's latent spaces, users can apply standard GAN editing vectors (e.g., aging, adding a smile) concurrently with the re-enactment process.
Discussion and Limitations
While the framework sets a new standard for high-resolution re-enactment, it inherits certain StyleGAN2 characteristics. The most prominent is "texture sticking," where fine details like skin pores or beard textures appear fixed to the screen rather than moving naturally with the face. Additionally, like most GAN-based approaches, the model struggles with heavy occlusions (e.g., hands in front of the face) and complex backgrounds.
Future work may involve adapting this framework to StyleGAN3, which is designed to be more translation-invariant and could potentially mitigate the texture-sticking issue.
Summary updated