Abstract: Learning the distribution of multi-object scenes with Generative Adversarial Networks (GAN) is challenging. Guiding the learning using semantic intermediate representations, which are less complex than images, can be a solution. In this article, we investigate splitting the optimisation of generative adversarial networks into two parts, by first generating a semantic segmentation mask from noise and then translating that segmentation mask into an image. We performed experiments using images from the CityScapes dataset and compared our approach to Progressive Growing of GANs (PGGAN), which uses multiscale growing of networks to guide the learning. Using the lens of a segmentation algorithm to examine the structure of generated images, we find that our method achieves higher structural consistency in latent space interpolations and yields generations with better differentiation between distinct objects, while achieving the same image quality as PGGAN as judged by a user study and a standard GAN evaluation metric.
The Challenge of Multi-Object Scenes
Generative Adversarial Networks (GANs) have become incredibly proficient at creating high-resolution images of single, centered objects, such as faces. However, they often struggle when tasked with complex, multi-object scenes like urban city streets.
A primary failure mode identified in models like Progressive Growing GANs (PGGAN) is the lack of internal understanding of discrete objects. While these models can mimic "scene-like" textures, they often produce what can be described as "texture mush." During latent space interpolations—the smooth transitions between two generated images—objects do not move or change naturally; instead, textures simply morph into one another. For instance, a car might dissolve into a sidewalk because the model does not recognize them as separate entities. This lack of global structural coherence prevents the generated scenes from feeling like a physical, structured world.
A Two-Stage Generative Pipeline
To address these structural deficiencies, this research proposes a "divide and conquer" approach that decomposes the generation process into two distinct stages:
- Layout Prediction: A GAN (specifically a Wasserstein GAN with Gradient Penalty) is trained to generate semantic segmentation masks from random noise. These masks are color-coded maps where each color represents a specific class (e.g., car, road, tree). Because these layouts are structurally simpler than full RGB images, the GAN can more easily learn the fundamental rules of scene composition.
- Conditional Synthesis: Once a layout is established, a second model—pix2pix—is used to translate that mask into a realistic RGB image. This stage focuses entirely on texturing the provided layout.
By separating where things are from what they look like, the system gains a crude but effective "world model." This "semantic bottleneck" forces the generative process to commit to a layout before worrying about pixel-level details, preventing the model from using ambiguous textures to bridge the gap between different object classes.
Experimental Results and the "Lens of Segmentation"
To evaluate the structural integrity of the models, the researchers utilized the CityScapes dataset. A key innovation in their evaluation was the use of a pre-trained segmentation tool (DeepLab) as a "lens" to examine the generated images. When applied to PGGAN's outputs, the resulting segmentations were chaotic and flickered wildly during transitions. In contrast, the proposed two-stage method maintained distinct, recognizable objects throughout the interpolation process.
Key Performance Metrics:
- Image Quality: In a blind user study, participants rated the realism of the proposed method as equal to the PGGAN baseline.
- Quantitative Performance: Using the Fréchet Inception Distance (FID) metric, the proposed method achieved a score of 56.37 (using coarse annotations), outperforming the PGGAN baseline of 63.90.
- Structural Consistency: Qualitatively, the proposed method showed much higher consistency. Objects appeared to move, grow, or shrink logically during transitions rather than fading in and out of existence.
Discussion and Future Directions
While the decomposition approach yields superior structural results, it is not without limitations. The method is notably "data hungry," relying heavily on labeled semantic data. When trained on a smaller set of "fine" annotations (3,000 images), the FID score worsened significantly to 96.33, indicating that the layout generator requires substantial samples to learn complex shapes.
Future work could explore training the layout generator and texture synthesizer jointly (end-to-end) or incorporating 3D geometry and depth maps to move beyond 2D layouts toward a more robust 3D understanding of the world.
Summary updated