LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis

Jiapeng Zhu, Ceyuan Yang, Yujun Shen, Zifan Shi, Bo Dai, Deli Zhao, Qifeng Chen

TL;DR

LinkGAN introduces an easy-to-use regularizer for GAN training that explicitly links specific axes of the latent space to designated pixel regions or semantic categories in the generated image. This allows for precise, localized image editing—such as closing a single eye or changing a car's color—simply by resampling the corresponding latent dimensions. Crucially, this method works for both 2D and 3D-aware GANs without requiring any architectural modifications.

What Problem Does This Paper Address?

While Generative Adversarial Networks (GANs) excel at producing photorealistic images, precisely controlling where changes occur during image manipulation remains a significant challenge. Most existing methods rely on posterior discovery, meaning researchers train a standard GAN first and then try to find meaningful directions within the high-dimensional latent space. This approach is often unstable, inaccurate, and inflexible.

Alternative methods achieve local control by directly modifying intermediate feature maps or completely redesigns the generator's architecture. However, these approaches sacrifice the simplicity of a clean latent space interface. The authors of LinkGAN tackle this gap by proposing a way to enforce a priori spatial or semantic control directly within the latent space during the training phase.

Methods

Instead of altering the neural network architecture, LinkGAN introduces a novel regularization loss during training.

The core idea is to partition the latent space $w$ into $K$ distinct subsets ($w = [w_1, ..., w_K]$) and map them directly to $K$ corresponding image pixel regions ($x = [x_1, ..., x_K]$). To enforce this explicit link, the authors use a dual-component perturbation loss:

  1. In-Region Protection Loss ($L_i$): Ensures that when you perturb the rest of the latent code ($w_i^c$), the pixels inside the target region ($x_i$) remain unchanged.
  2. Out-Region Protection Loss ($L_i^c$): Ensures that when you perturb the linked latent partition ($w_i$), the pixels outside the target region ($x_i^c$) remain unchanged.
Concept diagram of LinkGAN showing explicit link between latent axes and pixel regions

To keep training efficient, the authors employ a "lazy regularization" strategy, computing this regularizer only once every 8 iterations.

Datasets

The authors evaluated LinkGAN across several standard image synthesis benchmarks:

  • FFHQ: Human faces.
  • AFHQ: Animal faces.
  • LSUN-Church & LSUN-Car: Unaligned object categories.
  • LSUN-Bedroom: Indoor scenes (evaluated in the appendix).

For semantic-based control (e.g., isolating a car or the sky), they utilized an off-the-shelf segmentation model to dynamically generate pixel masks during training.

Key Insights and Ideas

  • Simplicity via Regularization: LinkGAN proves that localized control does not require complex architectural overhauls like block-wise latents or spatial transformers. A simple, well-formulated perturbation loss is sufficient.
  • Dynamic Semantic Mapping: The linked pixel region doesn't have to be a fixed bounding box. By incorporating a segmentation network, the regularizer can bind latent axes to dynamic semantic concepts (like "the sky"), adapting automatically even if the object moves or changes shape across different generated instances.
  • Geometry-Aware Control: When applied to 3D-aware GANs (like EG3D), LinkGAN automatically links the latent axes to both the 2D RGB appearance and the underlying 3D geometry. For example, modifying the "mouth" latent space opens the mouth in both the rendered image and the underlying 3D mesh.

Key Results

  • Precise Local Editing: LinkGAN significantly outperforms state-of-the-art baselines (StyleCLIP, ReSeFa, and StyleSpace) in preventing unwanted changes outside the edited zone. Quantitatively, when editing the eye region, LinkGAN achieved an out-of-region Mean Squared Error ($MSE_o$) of 2.24, compared to StyleCLIP's 74.17 and ReSeFa's 61.14 (lower is better).
  • Minimal Quality Trade-off: The regularizer preserves high image quality. On FFHQ using StyleGAN2, the Frechet Inception Distance (FID) only slightly shifted from 3.98 (baseline) to 5.00 (LinkGAN).
  • Real Image Editability: The trained models remain fully compatible with GAN inversion techniques. Users can take a real photograph, invert it into the latent space, and cleanly manipulate specific regions independently.
Figure 1. Precise local control achieved by LinkGAN, where we can manipulate the image content within a spatial region (e.g., a single eye or the right half of the image) or a semantic category (e.g., car) simply by resampling the latent code on some sparse axes. Our approach works well for 2D image syntheses, like StyleGAN2 [27] (left three columns), and 3D-aware image synthesis, like EG3D [5] (right two columns). It is noteworthy that, under the 3D-aware case, we can control both the appearance and the underlying geometry.

Limitations and Future Work

Despite its success, the authors acknowledge a few limitations:

  • Minor Leakage: The latent-pixel linkage is not perfectly absolute; a tiny amount of visual leakage can still occur in unlinked regions during extreme edits.
  • Boundary Inconsistencies: Resampling a single latent partition can occasionally create visible seams or artifacts at the borders of the edited region. While the authors showed that involving the discriminator on perturbed images helps smooth these transitions, perfect blending remains an open challenge.
  • Diversity vs. Constraint: Enforcing strict spatial constraints inherently reduces the overall diversity (recall) of the generative model, a common trade-off when introducing controllability.

Summary updated

Abstract

This work presents an easy-to-use regularizer for GAN training, which helps explicitly link some axes of the latent space to a set of pixels in the synthesized image. Establishing such a connection facilitates a more convenient local control of GAN generation, where users can alter the image content only within a spatial area simply by partially resampling the latent code. Experimental results confirm four appealing properties of our regularizer, which we call LinkGAN. (1) The latent-pixel linkage is applicable to either a fixed region (\textit{i.e.}, same for all instances) or a particular semantic category (i.e., varying across instances), like the sky. (2) Two or multiple regions can be independently linked to different latent axes, which further supports joint control. (3) Our regularizer can improve the spatial controllability of both 2D and 3D-aware GAN models, barely sacrificing the synthesis performance. (4) The models trained with our regularizer are compatible with GAN inversion techniques and maintain editability on real images.
BibTeX
@inproceedings{zhu2023linkgan,
  author = {Jiapeng Zhu and Ceyuan Yang and Yujun Shen and Zifan Shi and Bo Dai and Deli Zhao and Qifeng Chen},
  title = {{LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure