Seq-SG2SL: Inferring Semantic Layout From Scene Graph Through Sequence to Sequence Learning

Boren Li, Boyu Zhuang, Mingyang Li, Jian GuICCV 2019 · DOI · arXiv

Abstract

Generating realistic image layouts from structured scene graphs is a critical step in automated image synthesis. Traditional methods using Graph Convolutional Networks (GCNs) often struggle with the complexity of large scenes. We present Seq-SG2SL, a framework that reframes layout generation as a sequence-to-sequence translation task. By linearizing scene graphs into "semantic fragments" and layouts into "brick-action codes," we leverage the Transformer architecture to achieve superior spatial reasoning. Our approach significantly outperforms GCN baselines and introduces SLEU, a novel metric for evaluating spatial layout accuracy.

Introduction

The generation of realistic images from textual descriptions remains a fundamental challenge in computer vision. A common pipeline involves first generating a semantic layout—a set of bounding boxes defining object placement—before synthesizing the final pixels. While scene graphs provide a rich, structured input for this task, existing approaches based on Graph Convolutional Networks (GCNs) face a "combinatorial explosion" as the number of objects and relationships increases. GCNs attempt to process the entire scene holistically, which can become computationally prohibitive and spatially imprecise as scene complexity grows.

Seq-SG2SL addresses this by treating layout generation like building a structure from a blueprint. Instead of global optimization, the model learns the fundamental "actions" required to place objects based on their local relationships, effectively treating the problem as a machine translation task.

Methods

The Seq-SG2SL (Sequence-Scene Graph to Semantic Layout) framework utilizes a sequence-to-sequence proxy to bridge the gap between graphs and layouts.

Sequence Proxies

The core innovation lies in the linearization of data into two primary sequences:

  1. Input (SF): The scene graph is decomposed into a sequence of Semantic Fragments (SF). Each fragment is a triplet consisting of a subject, a predicate, and an object (e.g., man, riding, bike).
  2. Output (BACS): The layout is represented as a sequence of Brick-Action Code Segments (BACS). Each BACS is a 10-token "sentence" defining the class, position, and scale of the objects within a relationship.
The Seq-SG2SL pipeline: from scene graph to semantic layout via sequence-to-sequence learning.

Relative Position Encoding and Architecture

To better capture spatial semantics, BACS encodes object positions relative to their subjects. This allows the model to learn the geometric meaning of predicates like "above" or "inside" more effectively than absolute coordinate systems. The architecture employs a standard Transformer with six layers of self-attention to translate SF sequences into BACS sequences. This sequential approach allows the model to focus on local dependencies rather than solving the entire global geometry at once.

SLEU Metric

The authors introduce SLEU (Semantic Layout Evaluation Understudy), inspired by the BLEU metric in natural language processing. SLEU evaluates predicted relationships against ground truth using the Jaccard Index (IoU) to accommodate spatial tolerances, providing a more nuanced measure of layout accuracy than simple coordinate matching.

Results and Discussion

The model was evaluated on the Visual Genome (VG) dataset, filtered to 178 object classes and 45 relationship types to ensure data consistency.

Quantitative Performance

Seq-SG2SL demonstrated a substantial improvement over GCN-based baselines. On the training set, it achieved a Mean-SLEU (at IoU 0.5) of 0.4174, compared to the baseline's 0.0217. On the test set, it maintained a lead with a score of 0.0344 vs. 0.0205. This suggests that the sequential approach is far more expressive and capable of representing complex scene geometries than traditional graph-based methods.

Qualitative Analysis

Visual results demonstrate the model's ability to generate coherent layouts that respect the input scene graph's constraints.

Qualitative results showing generated layouts and their corresponding SLEU scores across various scene complexities.

Ablation studies confirmed that relative position encoding is vital; removing it led to a significant drop in performance, highlighting the importance of "causal" spatial reasoning in scene construction.

Conclusion

Seq-SG2SL demonstrates that treating spatial layout generation as a language translation task is a powerful alternative to graph-based methods. While the current model shows high capacity, limitations remain, such as the "single-reference trap" in evaluation—where multiple valid layouts might exist for one graph—and the need to integrate object attributes like color or size. Future work will focus on more sophisticated object merging strategies and multi-reference evaluation to better capture the inherent diversity of valid scene layouts.

Summary updated

BibTeX
@inproceedings{li2019seq,
  author = {Boren Li and Boyu Zhuang and Mingyang Li and Jian Gu},
  title = {{Seq-SG2SL: Inferring Semantic Layout From Scene Graph Through Sequence to Sequence Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00753},
}
Figure