A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing

Gwilherm Lesné, Yann Gousseau, Saïd Ladjal, Alasdair Newson

TL;DR

Researchers have developed a lightweight auto-encoder that re-organizes StyleGAN’s latent space into a "slider-friendly" format where each axis controls a specific, independent attribute (like age or expression). By training entirely within a compressed latent sub-space and enforcing decorrelation, the method achieves superior disentanglement and image fidelity with a remarkably short training time of just 45 minutes.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs), specifically StyleGAN, are world-class at creating photorealistic faces. However, "editing" these faces is notoriously difficult because their internal mathematical representations (latent spaces) are entangled.

If you try to move a point in the latent space to make a face look older, you might accidentally change the person's gender, add glasses, or alter the background. To be useful for digital artists, an editing tool needs three things:

  1. Disentanglement: Changing one attribute shouldn't affect others.
  2. Controllability: Attributes should map directly to intuitive axes (like a slider).
  3. Fidelity: The person’s identity and the image's overall quality should remain intact.

Existing methods often require complex optimization or move into expanded latent spaces ($W+$) that can lead to "unnatural" images that don't look like the original model's output.

Methods

The authors propose a simple yet effective auto-encoder architecture ($E$ and $D$) that sits on top of StyleGAN2. Their approach follows three key steps:

  • PCA Compression: They observed that StyleGAN’s latent space ($W$) is over-parametrized. By applying Principal Component Analysis (PCA), they found that just 60 dimensions (out of 512) capture 80% of the variance. They perform all editing within this "safe" 60-dimensional sub-space ($W_{PCA}$), which helps maintain image fidelity.
  • The Semantic Auto-encoder: The encoder $E$ maps $W_{PCA}$ to a new space $C$. In this space, the first $K$ dimensions are explicitly forced to match specific attributes (e.g., dimension 1 = smile, dimension 2 = age).
  • The Loss Functions: To train the model, they use a three-part loss:
  • Reconstruction Loss: Ensures the auto-encoder can return to the original vector if no edits are made.
  • Attribute Loss: Forces the latent axes to match labels from a pre-trained classifier.
  • Correlation Loss: This is the "secret sauce." It penalizes correlation between the latent axes, forcing the model to find truly independent (disentangled) directions for each attribute.

Crucially, the training happens entirely in the latent space. Unlike many other methods, they don't need to generate images during the training loop, which drastically reduces computational cost.

Datasets

The authors used a database of 200,000 synthetic latent vectors generated by StyleGAN2. To label these images for training, they used an EfficientNet-B0 classifier pre-trained on the CelebA dataset, covering 40 different facial attributes.

Key Insights and Ideas

  • Less is More: By restricting the editing to the top 60 PCA components, the model avoids "low-variance" regions of the latent space that typically cause visual artifacts or "ghosting" during edits.
  • Latent-Only Training: Because the loss functions operate on vectors rather than pixels, the training is incredibly efficient. It requires no forward passes through the heavy StyleGAN generator, making it accessible to researchers without massive GPU clusters.
  • The "Slider" Philosophy: By mapping attributes directly to axes, they transform a complex high-dimensional navigation problem into a simple linear adjustment, mimicking the workflow of professional photo editing software.

Key Results

  • Speed: The model trains in approximately 45 minutes on a single Nvidia RTX 3090.
  • Disentanglement: Quantitative "attribute variation matrices" show that their method (Variant C) has significantly lower off-diagonal interference than state-of-the-art competitors like InterfaceGAN and GANSpace.
  • Fidelity: The method achieved higher Identity Preservation scores (using cosine distance of face embeddings) and better FID (Fréchet Inception Distance) scores than competing methods, meaning the edited images look more like the original person and maintain higher visual quality.
  • Qualitative Success: Visual results demonstrate that "Age" edits in this model don't accidentally trigger "Smile" or "Gender" changes, a common failure point in previous work.

Limitations and Future Work

  • Classifier Dependency: The system is only as good as the attribute classifier used to label the training data. If the classifier is biased or inaccurate, the editing axes will be too.
  • Attribute Limit: The current study focused on 40 attributes. While comprehensive for faces, expanding this to more granular or non-facial domains remains a challenge.
  • Native Space Constraints: By staying in the native $W$ space for better fidelity, the model might have slightly less "editing reach" than methods that use the expanded $W+$ space, though the authors argue this is a worthwhile trade-off for realism.

Summary updated

Abstract

Recent advances in the field of generative models and in particular generative adversarial networks (GANs) have lead to substantial progress for controlled image editing, especially compared with the pre-deep learning era. Despite their powerful ability to apply realistic modifications to an image, these methods often lack properties like disentanglement (the capacity to edit attributes independently). In this paper, we propose an auto-encoder which re-organizes the latent space of StyleGAN, so that each attribute which we wish to edit corresponds to an axis of the new latent space, and furthermore that the latent axes are decorrelated, encouraging disentanglement. We work in a compressed version of the latent space, using Principal Component Analysis, meaning that the parameter complexity of our autoencoder is reduced, leading to short training times ($\sim$ 45 mins). Qualitative and quantitative results demonstrate the editing capabilities of our approach, with greater disentanglement than competing methods, while maintaining fidelity to the original image with respect to identity. Our autoencoder architecture simple and straightforward, facilitating implementation.
BibTeX
@inproceedings{lesne2023compact,
  author = {Gwilherm Lesné and Yann Gousseau and Saïd Ladjal and Alasdair Newson},
  title = {{A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing}},
  booktitle = {CVMP},
  year = {2023},
  url = {https://openreview.net/forum?id=ofoYjzkAeX},
}

Loading related content...

Figure