Transformer-based image generation from scene graphs

Renato Sortino, Simone Palazzo, Francesco Rundo, Concetto Spampinato

Introduction

Generating realistic images from structured descriptions like scene graphs is a significant challenge in computer vision. While text-to-image models offer broad descriptive power, scene graphs—where nodes represent objects and edges define spatial relationships (e.g., "left of", "above")—provide much more granular control over image composition. Traditional methods typically rely on Graph Convolutional Networks (GCNs) for layout prediction and Generative Adversarial Networks (GANs) for rendering. However, GANs are often unstable, prone to mode collapse, and struggle with complex global dependencies.

Methodology

This paper proposes a fully transformer-based pipeline that treats image generation as a sequence prediction task, ensuring greater training stability and diversity.

The proposed two-stage transformer architecture: SGTransformer for layout prediction and an Image Transformer for autoregressive synthesis.

The architecture consists of two primary stages:

  1. SGTransformer (Scene Graph to Layout): This module uses a transformer-based encoder with Laplacian positional encoding to capture the graph's topology. It predicts a layout comprising bounding boxes and object labels. The SGTransformer achieved a mean Intersection over Union (mIoU) of 42.03 on COCO, significantly outperforming GCN-based methods in object placement.
  2. Image Transformer (Layout to Image): A Vector-Quantized Variational Autoencoder (VQVAE) first compresses images into discrete visual tokens. An autoregressive transformer then predicts these tokens sequentially. The predicted layout is converted into triplets (class, top-left, bottom-right) and prepended to the image token sequence. The authors found that this unified self-attention approach outperformed cross-attention, allowing the model to attend to spatial constraints throughout the entire generation process.

Key Results

The model was evaluated on COCO-Stuff and Visual Genome (VG) benchmarks. Quantitatively, it achieved state-of-the-art Inception Scores (13.7 on COCO and 12.8 on VG), indicating superior image quality and diversity compared to previous methods like SG2IM and PasteGAN.

Qualitative comparison showing the proposed method's ability to generate more coherent and detailed images from complex scene graphs compared to baselines.

Qualitatively, the transformer-based approach demonstrates significantly better structural coherence. Unlike GANs, which often produce blurry or distorted objects in complex scenes, the autoregressive sampling allows for high diversity; multiple distinct, valid images can be generated from a single scene graph by sampling different token sequences. While the VQVAE's compression can slightly penalize FID scores compared to GANs that focus on high-frequency textures, the visual results show much clearer object definitions.

Conclusion

By moving away from adversarial training and adopting a unified transformer architecture, this work achieves significant improvements in layout accuracy and image synthesis. While computationally more intensive than GANs due to token-by-token generation, the benefits in stability and compositional control mark a significant step forward for scene-graph-to-image generation.

Summary updated

BibTeX
@article{sortino2023transformer,
  author = {Renato Sortino and Simone Palazzo and Francesco Rundo and Concetto Spampinato},
  title = {{Transformer-based image generation from scene graphs}},
  journal = {Comput. Vis. Image Underst.},
  year = {2023},
  doi = {10.1016/j.cviu.2023.103721},
}
Figure