Semantic Bottleneck Scene Generation

Samaneh Azadi, Michael Tschannen, Eric Tzeng, Sylvain Gelly, Trevor Darrell, Mario LucicarXiv 2019 · arXiv

Abstract

While Generative Adversarial Networks (GANs) excel at creating realistic faces or isolated objects, synthesizing complex scenes with multiple interacting elements remains a formidable task. Unconditional GANs frequently produce images with local texture realism but global structural failures, such as floating cars or nonsensical architecture. We present Semantic Bottleneck GAN (SB-GAN), a framework that decouples global scene structure from local appearance. By first generating a semantic segmentation map and then synthesizing the final image conditioned on that map, SB-GAN ensures structural integrity. Our approach achieves state-of-the-art performance on the Cityscapes and ADE-Indoor datasets, demonstrating superior global coherence and image quality compared to existing unconditional models.

Introduction

The evolution of GANs has moved from simple digit generation to high-resolution human faces. However, "scene synthesis"—the generation of complex environments like streetscapes or indoor rooms—is significantly harder. In these environments, the spatial relationship between objects (e.g., a car must be on a road, a bed must be on a floor) is as important as the texture of the objects themselves.

Traditional unconditional GANs attempt to learn both structure and texture simultaneously from a single noise vector, often failing to maintain a coherent "plan." Conversely, conditional models like SPADE produce excellent results but require human-provided semantic maps as input. SB-GAN bridges this gap by "dreaming" its own plan before painting the details.

Methodology: The Semantic Bottleneck

The core innovation of SB-GAN is the Semantic Bottleneck, a two-stage pipeline that factorizes the generation task.

The SB-GAN architecture uses a semantic bottleneck to separate the generation of scene structure from the rendering of local textures.

1. Segmentation Synthesis

The model begins by generating a semantic segmentation map from a random noise vector $z$. This map acts as a pixel-wise label guide (e.g., identifying regions as "sky," "building," or "car"). This stage utilizes a Progressive GAN (ProGAN) architecture. To handle the discrete nature of category labels while maintaining differentiability for end-to-end training, the authors employ the Gumbel-softmax trick.

2. Image Synthesis

The generated segmentation map is passed to a conditional synthesis network based on the SPADE architecture. This network is responsible for "rendering" the scene, adding realistic textures, lighting, and shadows based on the layout provided by the bottleneck.

3. End-to-End Fine-tuning

A critical component of the training process is the joint optimization of both stages. An additional unconditional discriminator evaluates the final RGB image, providing feedback that forces the segmentation generator to produce layouts that are not just plausible as maps, but are specifically optimized for high-quality photographic rendering.

Experimental Results

The researchers evaluated SB-GAN on three benchmarks: Cityscapes-5K, Cityscapes-25K, and the highly diverse ADE-Indoor dataset.

Quantitative Performance

SB-GAN demonstrated significant improvements in Frechet Inception Distance (FID), a standard metric for evaluating the quality and diversity of generated images. On the Cityscapes-5K dataset, SB-GAN achieved an FID of 65.49, vastly outperforming ProGAN's 92.57.

Qualitative Comparison and Human Evaluation

Qualitatively, SB-GAN produces scenes with far greater structural logic than its predecessors. While models like BigGAN often "collapse" when faced with the high diversity of indoor scenes, SB-GAN remains stable, producing recognizable rooms with coherent furniture arrangements.

A comparison of unconditional scene generation results. SB-GAN maintains superior global coherence and sharper object boundaries compared to ProGAN and BigGAN.

In Amazon Mechanical Turk user studies, evaluators consistently preferred SB-GAN's outputs, citing better structural integrity and overall realism.

Discussion and Limitations

The "divide and conquer" strategy of the semantic bottleneck effectively addresses the global coherence problem. By training on its own generated (and sometimes imperfect) layouts, the image synthesis stage becomes more robust than models trained only on ground-truth data.

However, the model's reliance on semantic labels during training is a limitation; it is not fully unsupervised. Furthermore, managing the complexity of discrete label generation remains a technical challenge. Future work may explore unsupervised bottlenecks that learn latent structural representations without human-defined categories.

Conclusion

SB-GAN represents a significant step forward in unconditional scene synthesis. By introducing a semantic bottleneck, the model gains the ability to "plan" its compositions, leading to images that are not only textured realistically but are structurally sound. This approach paves the way for more controllable and coherent generative models in complex domains.

Summary updated

Abstract

Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them to learn the scene structure. During inference, our model first synthesizes a realistic segmentation layout from scratch, then synthesizes a realistic scene conditioned on that layout. For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts. For the latter, we use a conditional segmentation-to-image synthesis network that captures the distribution of photo-realistic images conditioned on the semantic layout. When trained end-to-end, the resulting model outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and user-study evaluations. Moreover, we demonstrate the generated segmentation maps can be used as additional training data to strongly improve recent segmentation-to-image synthesis networks.
BibTeX
@misc{azadi2019semantic,
  author = {Samaneh Azadi and Michael Tschannen and Eric Tzeng and Sylvain Gelly and Trevor Darrell and Mario Lucic},
  title = {{Semantic Bottleneck Scene Generation}},
  year = {2019},
  eprint = {1911.11357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.11357},
}
Figure