A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing

Gwilherm Lesné, Yann Gousseau, Saïd Ladjal, Alasdair Newson

Abstract

Recent advances in generative adversarial networks (GANs), particularly StyleGAN, have enabled significant progress in photorealistic image generation. However, these models often lack disentanglement—the ability to edit specific attributes independently without affecting others. We propose a lightweight auto-encoder architecture that re-organizes the StyleGAN latent space into a compact, semantic representation where each axis corresponds to a specific, decorrelated attribute. By operating within a compressed latent subspace derived via Principal Component Analysis (PCA), our method reduces parameter complexity and achieves remarkably short training times (~45 minutes). Quantitative and qualitative evaluations demonstrate that our approach provides superior disentanglement and identity preservation compared to existing state-of-the-art methods.

Introduction

Generative Adversarial Networks (GANs) have set the standard for high-fidelity image synthesis. Despite their power, navigating their internal latent spaces for precise image editing remains a challenge. In StyleGAN, the latent space $W$ is often highly entangled; for instance, attempting to increase a subject's "age" might inadvertently alter their gender or background.

To be effective for practical applications, an image editing framework must satisfy three primary criteria:

  1. Disentanglement: Modifications to one attribute should not leak into others.
  2. Controllability: Attributes should map to intuitive, linear axes (e.g., sliders).
  3. Fidelity: The edited image must maintain the original subject's identity and overall visual quality.

While existing methods attempt to find directions in the latent space for editing, they often require complex optimization or utilize expanded spaces (like $W+$) that can introduce unnatural artifacts. Our work addresses these issues by introducing a semantic auto-encoder that enforces independence between attributes directly within a compressed latent space.

Methodology

PCA-Based Compression

The latent space $W$ of StyleGAN2 is typically 512-dimensional and over-parametrized. We observed that a significant portion of the variance is captured by a small number of dimensions. By applying Principal Component Analysis (PCA), we identified that just 60 dimensions ($W_{PCA}$) capture approximately 80% of the variance. Restricting editing to this "safe" subspace helps prevent the model from entering low-variance regions that typically cause "ghosting" or visual artifacts.

Semantic Auto-encoder Architecture

We introduce an encoder $E$ and a decoder $D$ that sit atop StyleGAN2. The encoder maps $W_{PCA}$ to a new semantic space $C$. In this space, the first $K$ dimensions are explicitly mapped to specific attributes (e.g., dimension 1 for smile, dimension 2 for age). The decoder then maps these semantic coordinates back to the $W_{PCA}$ space for image generation.

Training and Loss Functions

The model is trained entirely within the latent space, avoiding the need for expensive image generation during the training loop. We utilize a composite loss function:

  • Reconstruction Loss: Ensures that the auto-encoder can accurately reconstruct the original latent vector when no edits are applied.
  • Attribute Loss: Uses labels from a pre-trained classifier to force specific latent axes to correspond to specific physical attributes.
  • Correlation Loss: This is the critical component for disentanglement. It penalizes correlations between different axes in the $C$ space, forcing the model to find independent directions for each attribute.

Experimental Setup

We utilized a dataset of 200,000 synthetic latent vectors generated by StyleGAN2. To provide ground-truth labels for training, we employed an EfficientNet-B0 classifier pre-trained on the CelebA dataset, covering 40 distinct facial attributes. The training process is highly efficient, requiring only 45 minutes on a single Nvidia RTX 3090 GPU.

Results and Discussion

Disentanglement and Fidelity

Quantitative analysis using attribute variation matrices shows that our method significantly reduces off-diagonal interference compared to competitors like InterfaceGAN and GANSpace. This indicates that our "Age" slider, for example, has a much lower impact on "Gender" or "Expression" than previous approaches.

Furthermore, our method achieves higher Identity Preservation scores (measured via cosine distance of face embeddings) and superior Fréchet Inception Distance (FID) scores. By staying within the native $W$ space and utilizing PCA compression, we ensure that edited images remain within the distribution of the original generator, preserving realism.

Qualitative Performance

Visual results confirm the quantitative findings. Edits performed using our semantic axes result in smooth, natural transitions. The "slider" philosophy—mapping attributes directly to independent axes—transforms complex high-dimensional navigation into a simple linear adjustment suitable for consumer-grade photo editing software.

Conclusion and Future Work

We have presented a simple yet effective method for creating a compact and semantic latent space for StyleGAN. By training a decorrelated auto-encoder on PCA-compressed latent vectors, we achieve high-quality, disentangled edits with minimal computational overhead.

Future work includes expanding this approach to more granular attributes beyond the 40 CelebA categories and exploring its application in non-facial domains. While our reliance on pre-trained classifiers introduces potential biases, the efficiency and fidelity of the current framework provide a robust foundation for controllable generative modeling.

Summary updated

Abstract

Recent advances in the field of generative models and in particular generative adversarial networks (GANs) have lead to substantial progress for controlled image editing, especially compared with the pre-deep learning era. Despite their powerful ability to apply realistic modifications to an image, these methods often lack properties like disentanglement (the capacity to edit attributes independently). In this paper, we propose an auto-encoder which re-organizes the latent space of StyleGAN, so that each attribute which we wish to edit corresponds to an axis of the new latent space, and furthermore that the latent axes are decorrelated, encouraging disentanglement. We work in a compressed version of the latent space, using Principal Component Analysis, meaning that the parameter complexity of our autoencoder is reduced, leading to short training times ($\sim$ 45 mins). Qualitative and quantitative results demonstrate the editing capabilities of our approach, with greater disentanglement than competing methods, while maintaining fidelity to the original image with respect to identity. Our autoencoder architecture simple and straightforward, facilitating implementation.
BibTeX
@inproceedings{lesne2023compact,
  author = {Gwilherm Lesné and Yann Gousseau and Saïd Ladjal and Alasdair Newson},
  title = {{A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing}},
  booktitle = {CVMP},
  year = {2023},
  url = {https://openreview.net/forum?id=ofoYjzkAeX},
}
Figure