Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation

Peiye Zhuang, Oluwasanmi Koyejo, Alexander G. Schwing

TL;DR

Researchers from UIUC have developed a framework for semantic image editing that treats a Generative Adversarial Network's (GAN) latent space like a navigable map. By training multiple attribute transformations simultaneously and integrating a dedicated attribute regressor, their method allows for continuous, "slider-like" control over features—such as age, weather, or expression—while strictly preserving the original image's identity and photo-realism.

The Challenge of Steering GANs

GANs are incredibly good at creating realistic images, but "steering" them to make specific edits is notoriously difficult. If you want to take a photo of a summer scene and make it look like winter, or add a smile to a portrait, you usually run into three major roadblocks:

  1. Entanglement: Changing one attribute (e.g., adding "age") accidentally changes another (e.g., adding "glasses").
  2. Identity Loss: The edited person or scene stops looking like the original; the core "soul" or structure of the image is lost.
  3. Binary Constraints: Many existing methods only support "on/off" switches rather than smooth, continuous transitions.

The authors aim to move beyond simple image-to-image translation toward a more granular, controllable navigation of the GAN's internal "latent space."

The core idea is to find specific directions ($d$) in the GAN’s latent space that correspond to semantic attributes. To do this effectively, the authors introduce several key components:

  • Latent Space Navigation: Instead of modifying pixels directly, they modify the latent vector $z$. An edited image is generated from $z' = z + \sum \epsilon_i d_i$, where $\epsilon$ is the "strength" of the edit.
  • Attribute Regressor ($R$): They use a pre-trained model to predict attribute values from generated images. This regressor acts as a supervisor during training, ensuring that moving in direction $d$ actually results in the intended visual change.
  • Joint Sampling Strategy: Unlike prior work that looked at attributes in isolation (e.g., "clouds" vs. "no clouds"), this method samples multiple attributes simultaneously. This "joint" approach is the secret sauce for disentanglement—it teaches the model that the direction for "smiling" should be mathematically independent of the direction for "hair color."
  • Triple-Loss Objective:
    1. Regression Loss: Ensures the edit matches the desired intensity.
  • Discriminator Loss: Keeps the edited image looking like a real photo.
  • Content/Perceptual Loss: Compares high-level features of the original and edited images to ensure the identity (the specific face or landscape structure) remains unchanged.
  • Global vs. Local Transformations: While "Global" directions apply the same shift to every image, the authors also propose "Local" transformations. These use a small Multi-Layer Perceptron (MLP) to calculate a custom direction based on the specific input image, which helps in complex cases where a one-size-fits-all direction fails.

Key Insights

  • Disentanglement through Joint Training: The most significant breakthrough is that training for multiple attributes at once is better than training for them individually. By seeing all attributes vary simultaneously, the optimization process learns to "clean" the directions, removing unwanted side effects.
  • The Regressor as a Continuous Compass: By using a regressor that outputs a continuous range (0 to 1) rather than a binary label, the model learns to treat attributes as gradients. This enables the intuitive "slider" functionality for human editors.
  • Identity is a Feature, Not a Pixel: By using a perceptual loss (comparing deep network activations) rather than a pixel-wise loss, the model understands that a person's identity is about their structure and features. This allows the lighting or expression to change while the person remains highly recognizable.

Results and Performance

The model was evaluated on human faces (FFHQ, CelebA, CelebA-HQ) and natural scenes (Transient Attribute Database, MIT Places2), comparing against heavyweights like InterFaceGAN and Ganspace.

Qualitative comparison of image editing showing the proposed method against baselines for both scenes and faces.
  • Qualitative Improvements: The proposed method successfully adds smiles or changes seasons without the "ghosting" artifacts or identity shifts seen in baseline methods. For example, when adding "age," baselines often accidentally added glasses; this model successfully disentangles the two.
  • Quantitative Success: In user studies, participants rated the model's edits as having higher attribute presence and better identity preservation across categories like "clouds," "night," and "snow."
  • Numerical Disentanglement: The authors measured how much "other" attributes changed when editing a target attribute. Their method showed significantly lower "leakage" (e.g., editing "smile" resulted in much smaller unintended changes to "hair color" compared to prior state-of-the-art).

Limitations and Future Directions

While highly effective, the framework does have limitations. The quality of the edits is capped by the quality of the underlying GAN and the accuracy of the attribute regressor; if the regressor is biased, the edits will be too. Furthermore, while the model navigates the latent space well, it cannot create features that the original GAN never learned (e.g., adding a hat if the GAN was only trained on hatless faces).

Future directions include applying these latent navigation techniques to 3D controllable image editing and improving the interpretability of the learned mappings. The authors also acknowledge the potential for misuse in creating "deepfakes" and emphasize that their work should be paired with research into edited-image detection.

Summary updated

Abstract

Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of transformation functions, and apply a content loss and an adversarial loss that encourages the maintenance of image identity and photo-realism. We propose quantitative evaluation strategies for measuring controllable editing performance, unlike prior work, which primarily focuses on qualitative evaluation. Our model permits better control for both single- and multiple-attribute editing while preserving image identity and realism during transformation. We provide empirical results for both natural and synthetic images, highlighting that our model achieves state-of-the-art performance for targeted image manipulation.
BibTeX
@inproceedings{zhuang2021enjoy,
  author = {Peiye Zhuang and Oluwasanmi Koyejo and Alexander G. Schwing},
  title = {{Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=KSGbhMtfZi},
}
Figure