A Latent Transformer for Disentangled Face Editing in Images and Videos

Xu Yao, Alasdair Newson, Yann Gousseau, Pierre Hellier

Abstract

High quality facial image editing is a challenging problem in the movie post-production industry, requiring a high degree of control and identity preservation. Previous works that attempt to tackle this problem may suffer from the entanglement of facial attributes and the loss of the person’s identity. Furthermore, many algorithms are limited to a certain task. To tackle these limitations, we propose to edit facial attributes via the latent space of a StyleGAN generator, by training a dedicated latent transformation network and incorporating explicit disentanglement and identity preservation terms in the loss function. We further introduce a pipeline to generalize our face editing to videos. Our model achieves a disentangled, controllable, and identity-preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos. We conduct extensive experiments on image and video datasets and show that our model outperforms other state-of-the-art methods in visual quality and quantitative evaluation.

Introduction

Facial attribute editing—such as adding a smile, changing age, or adding eyeglasses—is a staple of photo retouching and film post-production. However, automating this process with AI remains difficult due to two primary issues: entanglement and identity loss. Entanglement occurs when modifying one feature (e.g., gender) unintentionally alters another (e.g., hairstyle). Identity loss happens when the edit changes the person's unique facial structure, making them unrecognizable. While Generative Adversarial Networks (GANs) like StyleGAN produce high-quality images, navigating their internal "latent space" for precise, clean edits is a significant challenge, particularly for real-world photos and videos where temporal stability is required.

Methodology

The proposed framework centers on a Latent Transformer ($T$). A key technical insight of this work is that the common assumption of linear separability in latent space is often a limiting myth. Instead, the authors train a dedicated transformation network to find the optimal path in StyleGAN’s $W+$ latent space, which handles the complexities of real-face reconstruction with much higher fidelity.

Loss Functions

The model's precision is driven by two key technical innovations in its loss functions:

  1. Attribute Regularization ($L_{attr}$): This ensures the target attribute changes while others remain static. The loss is weighted based on attribute correlation (e.g., "chubby" and "double chin") to avoid over-constraining the model by forcing naturally related features apart.
  2. Latent Code Regularization ($L_{rec}$): This keeps the edited latent code close to the original, preserving the subject's unique identity.

Video Pipeline

For video applications, the researchers developed a pipeline involving face alignment, optical flow to mitigate landmark jitter, and Poisson blending to seamlessly integrate the edited face back into the original frames, ensuring temporal stability.

Key Results

The Latent Transformer was compared against state-of-the-art methods like GANSpace and InterFaceGAN. Quantitative evaluations showed higher attribute preservation rates and identity preservation scores across the board. Qualitatively, the model demonstrates superior disentanglement; for example, adding a beard does not inadvertently change skin tone or hair color.

Sequential facial attribute editing on real images of Emma Watson and Barack Obama using the Latent Transformer.

As shown in the figure above, the model can perform multiple, additive edits—such as adding a smile, bangs, and arched eyebrows—while keeping the base identity perfectly intact. The video pipeline successfully produces 1024x1024 resolution edits that remain temporally stable, avoiding the "flickering" common in frame-by-frame editing approaches.

Discussion and Limitations

While the model achieves state-of-the-art results, some limitations remain:

  • Extreme Poses: The model struggles with profile views, as the underlying StyleGAN and encoders are primarily trained on frontal faces.
  • Data Imbalance: Rare attributes (e.g., "wearing a hat") are harder for the transformer to learn realistically.
  • Stochastic Features: Features controlled by GAN "noise" (like specific hair strand placement) are difficult to edit precisely as they aren't fully captured in the latent code.

Conclusion

The Latent Transformer provides a robust solution for disentangled, identity-preserving facial editing. By leveraging the $W+$ latent space and sophisticated regularization that accounts for feature correlation, the method enables high-quality, additive edits on both static images and high-resolution video.

Summary updated

Abstract

High quality facial image editing is a challenging problem in the movie post-production industry, requiring a high degree of control and identity preservation. Previous works that attempt to tackle this problem may suffer from the entanglement of facial attributes and the loss of the person’s identity. Furthermore, many algorithms are limited to a certain task. To tackle these limitations, we propose to edit facial attributes via the latent space of a StyleGAN generator, by training a dedicated latent transformation network and incorporating explicit disentanglement and identity preservation terms in the loss function. We further introduce a pipeline to generalize our face editing to videos. Our model achieves a disentangled, controllable, and identity-preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos. We conduct extensive experiments on image and video datasets and show that our model outperforms other state-of-the-art methods in visual quality and quantitative evaluation. Source codes are available at https://github.com/InterDigitalInc/latent-transformer.
BibTeX
@inproceedings{yao2021latent,
  author = {Xu Yao and Alasdair Newson and Yann Gousseau and Pierre Hellier},
  title = {{A Latent Transformer for Disentangled Face Editing in Images and Videos}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=TPyWqrbg4vU},
}
Figure