Abstract
While Generative Adversarial Networks (GANs) excel at creating realistic faces or isolated objects, synthesizing complex scenes with multiple interacting elements remains a formidable task. Unconditional GANs frequently produce images with local texture realism but global structural failures, such as floating cars or nonsensical architecture. We present Semantic Bottleneck GAN (SB-GAN), a framework that decouples global scene structure from local appearance. By first generating a semantic segmentation map and then synthesizing the final image conditioned on that map, SB-GAN ensures structural integrity. Our approach achieves state-of-the-art performance on the Cityscapes and ADE-Indoor datasets, demonstrating superior global coherence and image quality compared to existing unconditional models.
Introduction
The evolution of GANs has moved from simple digit generation to high-resolution human faces. However, "scene synthesis"—the generation of complex environments like streetscapes or indoor rooms—is significantly harder. In these environments, the spatial relationship between objects (e.g., a car must be on a road, a bed must be on a floor) is as important as the texture of the objects themselves.
Traditional unconditional GANs attempt to learn both structure and texture simultaneously from a single noise vector, often failing to maintain a coherent "plan." Conversely, conditional models like SPADE produce excellent results but require human-provided semantic maps as input. SB-GAN bridges this gap by "dreaming" its own plan before painting the details.
Methodology: The Semantic Bottleneck
The core innovation of SB-GAN is the Semantic Bottleneck, a two-stage pipeline that factorizes the generation task.
1. Segmentation Synthesis
The model begins by generating a semantic segmentation map from a random noise vector $z$. This map acts as a pixel-wise label guide (e.g., identifying regions as "sky," "building," or "car"). This stage utilizes a Progressive GAN (ProGAN) architecture. To handle the discrete nature of category labels while maintaining differentiability for end-to-end training, the authors employ the Gumbel-softmax trick.
2. Image Synthesis
The generated segmentation map is passed to a conditional synthesis network based on the SPADE architecture. This network is responsible for "rendering" the scene, adding realistic textures, lighting, and shadows based on the layout provided by the bottleneck.
3. End-to-End Fine-tuning
A critical component of the training process is the joint optimization of both stages. An additional unconditional discriminator evaluates the final RGB image, providing feedback that forces the segmentation generator to produce layouts that are not just plausible as maps, but are specifically optimized for high-quality photographic rendering.
Experimental Results
The researchers evaluated SB-GAN on three benchmarks: Cityscapes-5K, Cityscapes-25K, and the highly diverse ADE-Indoor dataset.
Quantitative Performance
SB-GAN demonstrated significant improvements in Frechet Inception Distance (FID), a standard metric for evaluating the quality and diversity of generated images. On the Cityscapes-5K dataset, SB-GAN achieved an FID of 65.49, vastly outperforming ProGAN's 92.57.
Qualitative Comparison and Human Evaluation
Qualitatively, SB-GAN produces scenes with far greater structural logic than its predecessors. While models like BigGAN often "collapse" when faced with the high diversity of indoor scenes, SB-GAN remains stable, producing recognizable rooms with coherent furniture arrangements.
In Amazon Mechanical Turk user studies, evaluators consistently preferred SB-GAN's outputs, citing better structural integrity and overall realism.
Discussion and Limitations
The "divide and conquer" strategy of the semantic bottleneck effectively addresses the global coherence problem. By training on its own generated (and sometimes imperfect) layouts, the image synthesis stage becomes more robust than models trained only on ground-truth data.
However, the model's reliance on semantic labels during training is a limitation; it is not fully unsupervised. Furthermore, managing the complexity of discrete label generation remains a technical challenge. Future work may explore unsupervised bottlenecks that learn latent structural representations without human-defined categories.
Conclusion
SB-GAN represents a significant step forward in unconditional scene synthesis. By introducing a semantic bottleneck, the model gains the ability to "plan" its compositions, leading to images that are not only textured realistically but are structurally sound. This approach paves the way for more controllable and coherent generative models in complex domains.
Summary updated