A Layer-Based Sequential Framework for Scene Generation with GANs

Mehmet Ozgur Turkoglu, William Thong, Luuk J. Spreeuwers, Berkay KicanaogluAAAI 2019 · DOI

Introduction

Generating realistic, multi-object scenes is a significant challenge in computer vision. Most current state-of-the-art models, such as the original Pix2Pix, rely on "one-shot" generation. This approach maps a full semantic layout directly to a final image in a single pass. However, one-shot generation often suffers from a lack of user control, occlusion artifacts (like "ghosting" or merged shapes), and blurry results due to the complexity of learning an entire scene simultaneously.

The authors of this paper argue that a single noise vector cannot adequately capture the intricate details of a multi-entity scene. Instead, they propose a sequential, layer-based framework that mimics the workflow of a digital painter, adding elements one by one to build a complex image.

The proposed sequential generation process compared to one-shot baselines, showing step-by-step construction from background to multiple objects.

Methods

The core innovation is the Layer-Based Sequential Framework, which decomposes scene generation into specialized sub-tasks handled by two types of generators:

  • Background Generator ($G_{bg}$): This model creates the initial "canvas" (e.g., sky, grass, water). During training, it uses dual branches to understand the relationship between the empty background and the final intended scene.
  • Foreground Generator ($G_{fg}$): Acting as an "object stamper," this generator takes the current canvas, a specific object mask, and a noise vector defining the object's appearance. It treats the addition of an object as an inpainting task, filling in the masked area while maintaining consistency with the existing background's lighting and context.

To ensure high-quality output, the framework employs Dual Discriminators: a Global Discriminator to evaluate the overall image realism and a Local Discriminator to ensure sharp, realistic details on each newly added object.

Datasets and Evaluation

The framework was evaluated using the MS-COCO dataset, focusing on a subset of approximately 11,000 images across six semantically related categories: sheep, cow, bear, elephant, giraffe, and zebra. Images were resized to 128x128 pixels for the experiments.

Key Insights

  • The Painter’s Analogy: By mimicking a human painter who layers a tree over a pre-painted sky, the model handles occlusions naturally. The newest object simply overwrites the pixels behind it, avoiding the blending artifacts common in one-shot models.
  • Decoupling Content and Style: Using separate instance masks and noise vectors provides explicit control. Users can modify an object's appearance (style) via the noise vector or its position (content) via the mask without affecting the rest of the scene.

Key Results

The sequential model was compared against Standard CNNs, Pix2Pix, and Pix2Pix++.

  • Quantitative Performance: The proposed model achieved a Frechet Inception Distance (FID) of 28.7, significantly outperforming Pix2Pix (34.0). In terms of Semantic Accuracy (Mean IoU), the model scored 0.650, nearly doubling the standard CNN baseline and surpassing Pix2Pix++ (0.605).
  • Qualitative Superiority: As shown in the comparison below, the sequential approach maintains background integrity and object realism across various categories, whereas baselines often exhibit color bleeding or background distortion.
Qualitative comparison of the proposed sequential model against standard CNN, Pix2pix, and Pix2pix++ across various object categories.

Limitations and Future Work

While effective, the model currently depends on precise user-provided masks. Although the authors demonstrated mask generation from bounding boxes, this adds complexity to the pipeline. Additionally, the order of object placement can influence subtle details like shadow casting, suggesting that optimizing the generation sequence is a promising area for future research.

Summary updated

BibTeX
@inproceedings{turkoglu2019layer,
  author = {Mehmet Ozgur Turkoglu and William Thong and Luuk J. Spreeuwers and Berkay Kicanaoglu},
  title = {{A Layer-Based Sequential Framework for Scene Generation with GANs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018901},
}
Figure