StyleMapGAN: Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing

Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, Youngjung Uh

Generative Adversarial Networks (GANs), particularly StyleGAN, have set benchmarks in high-quality image synthesis. However, applying these models to real-world image editing remains a challenge due to the "GAN inversion" problem—the difficulty of mapping a real image back into the latent space. Existing methods typically rely on slow optimization or lossy encoders that compress spatial details into a 1D vector, leading to "entanglement" where changing one feature inadvertently alters others.

StyleMapGAN addresses these issues by replacing the traditional 1D latent vector ($w$) with a 3D Stylemap ($w \in \mathbb{R}^{C \times H \times W}$). This spatial representation allows the model to preserve local details and perform precise, localized edits in real-time.

Methodology: The Spatial Stylemap

The core innovation is the Stylemap, a 3D tensor that maintains explicit spatial dimensions. Unlike standard StyleGAN, which applies a global style via AdaIN, StyleMapGAN uses Spatially Variant Modulation. Here, the stylemap acts as a local controller, adjusting features at specific spatial locations.

To maintain this spatial information across resolutions, a Stylemap Resizer network scales the stylemap using convolutions and upsampling. The system is trained jointly with an encoder using adversarial, perceptual (LPIPS), and domain-guided losses to ensure high reconstruction fidelity and realistic outputs.

Key Insights: Localized Editing and High Fidelity

By maintaining spatial dimensions, StyleMapGAN avoids the "squashing" of local details. This leads to two major advantages:

  1. High Reconstruction Fidelity: The encoder can capture fine textures and shapes that are often lost in 1D latent spaces.
  2. Local Editing via Blending: Local manipulation becomes a straightforward alpha-blending operation in the latent space. For example, a specific architectural feature or facial attribute can be swapped by pasting the corresponding region of one stylemap into another.
Comparison of global vs. local editing for makeup and facial hair.

The authors found that an $8 \times 8$ stylemap resolution provides the optimal balance between reconstruction quality and natural integration of local edits. The model even handles "unaligned transplantation," where the generator naturally adjusts the size and pose of a transplanted feature (like a church tower) to fit its new surroundings.

Experimental Results

The model was evaluated on CelebA-HQ, AFHQ, and LSUN (Car and Church) datasets.

  • Speed: StyleMapGAN performs inversion and editing in under 0.1 seconds, offering a 100x to 2000x speedup over optimization-based methods.
  • Precision: The spatial nature of the stylemap allows for highly targeted edits using masks, as demonstrated in complex scenes like cars and buildings.
Spatial masks for local editing in complex scenes (LSUN Church and Car).

Limitations and Future Work

While powerful, StyleMapGAN struggles with extreme pose differences (e.g., profile vs. frontal views), which can cause ghosting artifacts. Additionally, significant size mismatches between reference and target features can lead to unnatural-looking transplantations. Future work may involve applying this spatial latent representation to other architectures like BigGAN or Variational Autoencoders (VAEs).

Conclusion

StyleMapGAN demonstrates that spatial dimensions are a critical "missing link" in GAN latent spaces. By moving beyond 1D vectors, it enables high-fidelity, real-time, and localized image editing across diverse domains.

Summary updated

BibTeX
@inproceedings{kim2021stylemapgan,
  author = {Hyunsu Kim and Yunjey Choi and Junho Kim and Sungjoo Yoo and Youngjung Uh},
  title = {{StyleMapGAN: Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=xSGqxE5r_av},
}
Figure