Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks

Chen Zhang, Yinghao Xu, Yujun ShenarXiv 2021 · arXiv

Introduction

Generative Adversarial Networks (GANs), particularly StyleGAN2, have achieved remarkable success in synthesizing high-quality, photo-realistic images. However, "steering" these models remains a significant challenge due to the entangled nature of their latent spaces. In complex scenes like bedrooms, attempting to change a single element—such as the color of a bed—often inadvertently alters unrelated parts of the image, like the wallpaper or window placement. This global entanglement prevents the fine-grained control necessary for practical design applications.

LoGAN (Local GAN) introduces a framework for precise, local control over image generation. By shifting manipulation from global latent codes to intermediate feature maps, LoGAN enables users to "re-decorate" rooms by removing, inserting, shifting, and rotating individual objects while maintaining seamless integration and perspective. This approach treats a generated image not as a flat grid of pixels, but as a collection of modifiable parts.

Methodology: Local Feature Manipulation

The core innovation of LoGAN lies in its use of intermediate feature maps, which encode rich spatial information. The framework employs several key components to achieve local control:

  • Content and Style Modulation (CMod & SMod): These operators allow for replacing features in specific masked regions or applying style transformations (via AdaIN) to local objects without affecting the global scene. This allows a user to change the texture of a bed without altering the rest of the room.
  • Priority Masking: To handle depth and occlusion (e.g., a lamp positioned in front of a window), a priority system determines which object's features "win" in the final composition.
  • Layout Parsing: The system predicts basic room geometry (walls, floor, ceiling) using a semantic segmentation model to create a "clean slate" for editing.
An empty bedroom base generated by predicting room geometry, serving as a canvas for local edits.

Key Insights: Layer Sensitivity

A critical finding of the LoGAN research is that different layers within the GAN generator are specialized for different types of edits. The researchers discovered that:

  1. Removal is most effective at early layers (e.g., Layer 4). Because these layers have a larger "receptive field," the generator can better "hallucinate" the background to fill the hole left by a removed object.
  2. Insertion is best performed at middle layers (e.g., Layer 7). This preserves the sharp details and textures of the new object without it being "eroded" by the surrounding background features.

By successfully generating a plausible empty room based on the original's geometry, the authors provide a stable foundation for complex multi-object editing that wasn't possible with previous global latent space methods.

Results and Multi-Object Composition

LoGAN demonstrates impressive qualitative results in seamless editing and multi-object composition. The system can remove a bed and replace it with a different one, or move a window across a wall, without creating visible "seams" or artifacts. Users can progressively build a room from scratch, adding furniture and decor that respect the room's perspective and lighting.

A complex bedroom scene synthesized through object-level composition, including furniture and wall art.

The system maintains style consistency across edits, ensuring that newly inserted objects match the existing lighting and aesthetic of the room. This level of control transforms the generative process from a "black box" into an interactive design tool.

Limitations and Future Work

Despite its strengths, LoGAN relies on the accuracy of external semantic segmentation models for layout parsing. Additionally, current object insertion requires "source" objects from the GAN's existing latent space rather than external user-provided images. Future research aims to extend this local control to other complex environments, such as kitchens or living rooms, and integrate these capabilities into user-friendly interior design software.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have made great success in synthesizing high-quality images. However, how to steer the generation process of a well-trained GAN model and customize the output image is much less explored. It has been recently found that modulating the input latent code used in GANs can reasonably alter some variation factors in the output image, but such manipulation usually presents to change the entire image as a whole. In this work, we propose an effective approach, termed as LoGAN, to support local editing of the output image. Concretely, we introduce two operators, i.e., content modulation and style modulation, together with a priority mask to facilitate the precise control of the intermediate generative features. Taking bedroom synthesis as an instance, we are able to seamlessly remove, insert, shift, and rotate the individual objects inside a room. Furthermore, our method can completely clear out a room and then refurnish it with customized furniture and styles. Experimental results show the great potentials of steering the image generation of pre-trained GANs for versatile image editing.
BibTeX
@misc{zhang2021decorating,
  author = {Chen Zhang and Yinghao Xu and Yujun Shen},
  title = {{Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks}},
  year = {2021},
  eprint = {2105.08222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.08222},
}
Figure