A Latent Transformer for Disentangled and Identity-Preserving Face Editing

Xu Yao, Alasdair Newson, Yann Gousseau, Pierre Hellier

Abstract: High quality facial image editing is a challenging problem in the movie post-production industry, requiring a high degree of control and identity preservation. Previous works that attempt to tackle this problem may suffer from the entanglement of facial attributes and the loss of the person's identity. Furthermore, many algorithms are limited to a certain task. To tackle these limitations, we propose to edit facial attributes via the latent space of a StyleGAN generator, by training a dedicated latent transformation network and incorporating explicit disentanglement and identity preservation terms in the loss function. We further introduce a pipeline to generalize our face editing to videos. Our model achieves a disentangled, controllable, and identity-preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos. We conduct extensive experiments on image and video datasets and show that our model outperforms other state-of-the-art methods in visual quality and quantitative evaluation.

Introduction

Facial attribute editing is a cornerstone of digital retouching and film post-production. However, performing these edits automatically on real-world images remains a significant technical challenge. Most existing tools struggle with two primary issues: entanglement and identity loss.

Entanglement occurs when facial features are coupled within the model's representation; for instance, attempting to change a subject's age might inadvertently alter their perceived gender or skin tone. Identity loss occurs when the editing process degrades the unique facial characteristics of the individual, resulting in a subject that no longer resembles the original person. While many generative models perform well on synthetic data, they often fail when applied to "in-the-wild" photos or high-resolution video where temporal consistency is paramount to avoid visual artifacts like flickering.

Methodology

This research leverages the structured latent space of StyleGAN2. To edit a real image, the system first "projects" the image into the extended latent space ($W+$), identifying the specific mathematical vector that represents the target face.

The core of the proposed architecture is the Latent Transformer ($T$). Unlike previous methods that search for global linear directions in latent space, this model trains a dedicated network to learn a non-linear transformation function. The effectiveness of the transformer is driven by a multi-component loss function:

  • Attribute Classification Loss ($L_{cls}$): Drives the target attribute toward the desired state (e.g., inducing a smile).
  • Attribute Regularization ($L_{attr}$): Acts as the disentanglement engine. It utilizes a pre-trained classifier to ensure that non-target attributes remain constant. A "correlation-weighted" term is included to prevent over-constraining features that are naturally statistically linked.
  • Identity Preservation ($L_{rec}$): Ensures the modified latent code remains in close proximity to the original, preserving the subject's unique identity.

For video applications, the pipeline incorporates face alignment, per-frame latent transformation, and Seamless Cloning (Poisson blending) to integrate edited faces back into original frames with high temporal stability.

Experimental Results

The model was evaluated using the CelebA-HQ and FFHQ datasets for training and image evaluation, while FILMPAC and RAVDESS were used to demonstrate performance on real-world video.

Quantitative Performance

When compared against state-of-the-art baselines like GANSpace and InterFaceGAN, the Latent Transformer consistently achieved higher attribute preservation rates and identity preservation scores. Specifically, when modifying complex attributes like age or gender, the model was significantly more effective at keeping unrelated features (such as eyeglasses or facial hair) unchanged.

Qualitative Performance

A key strength of the Latent Transformer is its ability to perform sequential editing. Because the model preserves identity and disentangles attributes so effectively, a single image can undergo multiple transformations in a row without the quality degrading or the identity shifting.

Figure 4: Sequential facial attribute editing on a real image of Barack Obama, demonstrating the addition of age, a smile, a beard, and eyeglasses while maintaining identity.
Figure 6: Sequential facial attribute editing on a real image of Emma Watson, showing the addition of a smile, bangs, arched eyebrows, and age.

All results are produced at a high resolution of 1024 x 1024, meeting the standards required for professional post-production.

Discussion and Insights

The Non-Linearity of Latent Space

The authors demonstrate that the assumption of linearity in latent space—a common hypothesis in earlier works—is a limiting factor for real-world image editing. The Latent Transformer's ability to navigate the $W+$ space non-linearly allows for much more accurate reconstructions and edits.

Temporal Stability

By operating within the latent space rather than at the pixel level, the model inherently maintains temporal stability in video. Because the latent codes of consecutive frames are naturally similar, the resulting edits do not suffer from the "jitter" or flickering common in other video editing techniques.

Limitations

Despite its success, the model faces challenges with extreme poses, such as side profiles. This is primarily a limitation of the underlying StyleGAN generator and the encoder used for projection; if the initial representation of the face is incomplete, the subsequent edits cannot be fully accurate. Additionally, attributes that are underrepresented in the training data (e.g., specific types of headwear) are more difficult for the transformer to learn effectively.

Conclusion

The Latent Transformer represents a significant step forward in disentangled facial editing. By combining the generative power of StyleGAN with a specialized transformation network and identity-preserving loss functions, the model provides a robust tool for high-resolution image and video manipulation that meets the rigorous demands of the media industry.

Summary updated

Abstract

High quality facial image editing is a challenging problem in the movie post-production industry, requiring a high degree of control and identity preservation. Previous works that attempt to tackle this problem may suffer from the entanglement of facial attributes and the loss of the person's identity. Furthermore, many algorithms are limited to a certain task. To tackle these limitations, we propose to edit facial attributes via the latent space of a StyleGAN generator, by training a dedicated latent transformation network and incorporating explicit disentanglement and identity preservation terms in the loss function. We further introduce a pipeline to generalize our face editing to videos. Our model achieves a disentangled, controllable, and identity-preserving facial attribute editing, even in the challenging case of real (i.e., non-synthetic) images and videos. We conduct extensive experiments on image and video datasets and show that our model outperforms other state-of-the-art methods in visual quality and quantitative evaluation. Source codes are available at https://github.com/InterDigitalInc/latent-transformer.
BibTeX
@inproceedings{yao2021latent,
  author = {Xu Yao and Alasdair Newson and Yann Gousseau and Pierre Hellier},
  title = {{A Latent Transformer for Disentangled and Identity-Preserving Face Editing}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=TPyWqrbg4vU},
}
Figure