Interactive Scene Generation via Scene Graphs with Attributes

Oron Ashual, Lior Wolf

Abstract

This paper introduces an interactive system for generating high-quality images from scene graphs by decoupling layout (spatial positioning) from appearance (visual style). Utilizing a dual-embedding architecture and novel discriminators, the model allows users to intuitively manipulate objects and swap styles independently. The resulting 256x256 images significantly outperform previous baselines in both resolution and visual fidelity.

Introduction

Generating realistic images from structured descriptions like scene graphs is a challenging "one-to-many" problem. Previous state-of-the-art methods often struggled with entanglement, where layout and appearance were baked together. For instance, changing the color of a car might inadvertently alter its shape or position. Furthermore, many existing systems were limited to low-resolution (64x64) outputs or required tedious manual masking.

This work aims to provide a user-friendly tool where a coarse layout can be provided via a scene graph, and the AI handles the complex geometry and textures while maintaining high visual fidelity and independent control over object attributes.

Proposed Architecture

The core innovation is a dual-embedding scheme processed through a composite neural network that separates spatial context from visual style.

The proposed architecture decouples layout and appearance embeddings to generate high-resolution images.

Dual-Embedding Pipeline

  1. Layout Embedding ($u_i$): A Graph Convolutional Network (GCN) processes the scene graph (nodes as objects, edges as relationships). This captures the object's context and relative position, predicting a bounding box ($b_i$).
  2. Appearance Embedding ($a_i$): A separate CNN extracts style features from reference images or learned archetypes.
  3. Mask Generation: A random noise vector ($z_i$) is added to the layout embedding to predict a detailed object mask ($m_i$), allowing for stochastic variety in shape.
  4. Rendering: A multiplexer combines these masks and appearance vectors into a high-dimensional tensor, which an encoder-decoder residual network ($R$) renders into the final image ($p$).

Key Insights and Appearance Transfer

By separating layout from appearance, the model can generate innumerable variations for a single scene graph. Users can keep the composition identical while cycling through different "archetypal" looks for objects.

Appearance transfer allows the style of a reference image to be applied to a generated object within a new scene.

A striking feature of this approach is appearance transfer. As shown above, the system can "copy-paste" the visual style of an object from a real photo (e.g., a specific bus texture) and apply it to a generated object in a completely different layout. This provides unprecedented creative control compared to previous end-to-end generation models.

Results and Discussion

The model achieves a significant resolution bump to 256x256 pixels, a fourfold increase over the primary baseline (Johnson et al., 2018). Qualitatively, the generated images are sharper and more recognizable, respecting spatial constraints like "person on sand" with higher accuracy.

While the system is highly interactive and fast, some limitations remain. Extremely complex physical interactions (e.g., "person sitting on a bicycle") can still be difficult for the GCN to translate into perfectly aligned masks. Additionally, the quality of generated styles depends on the diversity of the training data, though this is mitigated by the ability to use external reference images for style transfer.

Summary updated

Abstract

We introduce a simple yet expressive image generation method. On the one hand, it does not require the user to paint the masks or define a bounding box of the various objects, since the model does it by itself. On the other hand, it supports defining a coarse location and size of each object. Based on this, we offer a simple, interactive GUI, that allows a layman user to generate diverse images effortlessly.From a technical perspective, we introduce a dual embedding of layout and appearance. In this scheme, the location, size, and appearance of an object can change independently of each other. This way, the model is able to generate innumerable images per scene graph, to better express the intention of the user.In comparison to previous work, we also offer better quality and higher resolution outputs. This is due to a superior architecture, which is based on a novel set of discriminators. Those discriminators better constrain the shape of the generated mask, as well as capturing the appearance encoding in a counterfactual way.Our code is publicly available at https://www.github.com/ashual/scene_generation.
BibTeX
@inproceedings{ashual2020interactive,
  author = {Oron Ashual and Lior Wolf},
  title = {{Interactive Scene Generation via Scene Graphs with Attributes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i09.7112},
}
Figure