Abstract
This paper introduces an interactive system for generating high-quality images from scene graphs by decoupling layout (spatial positioning) from appearance (visual style). Utilizing a dual-embedding architecture and novel discriminators, the model allows users to intuitively manipulate objects and swap styles independently. The resulting 256x256 images significantly outperform previous baselines in both resolution and visual fidelity.
Introduction
Generating realistic images from structured descriptions like scene graphs is a challenging "one-to-many" problem. Previous state-of-the-art methods often struggled with entanglement, where layout and appearance were baked together. For instance, changing the color of a car might inadvertently alter its shape or position. Furthermore, many existing systems were limited to low-resolution (64x64) outputs or required tedious manual masking.
This work aims to provide a user-friendly tool where a coarse layout can be provided via a scene graph, and the AI handles the complex geometry and textures while maintaining high visual fidelity and independent control over object attributes.
Proposed Architecture
The core innovation is a dual-embedding scheme processed through a composite neural network that separates spatial context from visual style.
Dual-Embedding Pipeline
- Layout Embedding ($u_i$): A Graph Convolutional Network (GCN) processes the scene graph (nodes as objects, edges as relationships). This captures the object's context and relative position, predicting a bounding box ($b_i$).
- Appearance Embedding ($a_i$): A separate CNN extracts style features from reference images or learned archetypes.
- Mask Generation: A random noise vector ($z_i$) is added to the layout embedding to predict a detailed object mask ($m_i$), allowing for stochastic variety in shape.
- Rendering: A multiplexer combines these masks and appearance vectors into a high-dimensional tensor, which an encoder-decoder residual network ($R$) renders into the final image ($p$).
Key Insights and Appearance Transfer
By separating layout from appearance, the model can generate innumerable variations for a single scene graph. Users can keep the composition identical while cycling through different "archetypal" looks for objects.
A striking feature of this approach is appearance transfer. As shown above, the system can "copy-paste" the visual style of an object from a real photo (e.g., a specific bus texture) and apply it to a generated object in a completely different layout. This provides unprecedented creative control compared to previous end-to-end generation models.
Results and Discussion
The model achieves a significant resolution bump to 256x256 pixels, a fourfold increase over the primary baseline (Johnson et al., 2018). Qualitatively, the generated images are sharper and more recognizable, respecting spatial constraints like "person on sand" with higher accuracy.
While the system is highly interactive and fast, some limitations remain. Extremely complex physical interactions (e.g., "person sitting on a bicycle") can still be difficult for the GCN to translate into perfectly aligned masks. Additionally, the quality of generated styles depends on the diversity of the training data, though this is mitigated by the ability to use external reference images for style transfer.
Summary updated