R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion

Jinxiu Liu, Qi LiuAAAI 2024 · OpenReview

Abstract

Image generation tasks have achieved remarkable performance using large-scale diffusion models. However, these models are limited to capturing the abstract relations (viz., interactions excluding positional relations) among multiple entities of complex scene graphs. Two main problems exist: 1) fail to depict more concise and accurate interactions via abstract relations; 2) fail to generate complete entities. To address that, we propose a novel Relation-aware Compositional Contrastive Control Diffusion method, dubbed as R3CD, that leverages large-scale diffusion models to learn abstract interactions from scene graphs. Herein, a scene graph transformer based on node and edge encoding is first designed to perceive both local and global information from input scene graphs, whose embeddings are initialized by a T5 model. Then a joint contrastive loss based on attention maps and denoising steps is developed to control the diffusion model to understand and further generate images, whose spatial structures and interaction features are consistent with a priori relation. Extensive experiments are conducted on two datasets: Visual Genome and COCO-Stuff, and demonstrate that the proposal outperforms existing models both in quantitative and qualitative metrics to generate more realistic and diverse images according to different scene graph specifications.

Introduction: The Challenge of Abstract Relations

Scene Graph to Image Generation (SG2IM) is the task of turning a structured map of objects and their relationships into a realistic image. While current diffusion models are excellent at placing objects according to positional relations (e.g., "to the left of"), they frequently struggle with abstract relations—interactions that involve specific poses or semantic nuances, such as "shaking hands" or "riding."

Existing methods often suffer from two main issues:

  1. Relation Ambiguity: They fail to depict concise interactions, often resulting in objects that are near each other but not actually interacting.
  2. Entity Missing: In complex scenes with many objects, models frequently "forget" to render certain entities mentioned in the graph.
Figure 3: An illustration of relation ambiguity, where models fail to clearly define the interaction between entities.

Methodology: The R3CD Framework

The authors propose the R3CD framework, which introduces three key innovations to bridge the gap between symbolic graph edges and pixel-level features:

SGFormer (Scene Graph Transformer)

Instead of using standard Graph Convolutional Networks (GCNs) which only look at immediate neighbors, SGFormer uses a transformer architecture to capture both local and global information. It initializes node and edge embeddings using a pre-trained T5 model, allowing the system to leverage deep linguistic knowledge about the objects and actions.

Relation-aware Diffusion Contrastive Control

This is the core control mechanism, using two types of contrastive loss to teach the diffusion model what abstract relations look like:

  • Attention Map Contrastive Loss: Ensures that the spatial attention maps for similar actions look similar across different subjects, while remaining distinct from unrelated actions.
  • Diffusion Steps Contrastive Loss: Since the early stages of the diffusion process define high-level structure, the authors apply a contrastive loss to the predicted noise distributions. This aligns the model’s "internal blueprint" with the abstract relations defined in the graph.

Triplet-level Compositional Generation

Rather than generating the whole image as one unit, the model breaks the graph into triplets (Subject-Predicate-Object). It uses edge-embedding attention maps as layout guidance to ensure every interaction in the graph is represented in the final pixels.

Experimental Results

R3CD was trained and evaluated on two standard benchmarks: Visual Genome and COCO-Stuff. The model outperformed state-of-the-art models (like SGDiff) across both quantitative and qualitative metrics.

  • Quantitative: On the COCO-Stuff dataset, R3CD achieved an FID (Fréchet Inception Distance) of 32.9, a significant improvement over SGDiff’s 36.2. It also saw gains in Inception Score (IS), moving from 17.8 to 19.5.
  • Qualitative: Visual results show that R3CD is much better at rendering all entities in a complex graph. For example, when given the triplet (cat, ride on, dog), R3CD successfully rendered the complex interaction, whereas previous models often simply drew two animals standing near each other.
Figure 2: Qualitative result showing R3CD successfully rendering a complex 'cat riding dog' interaction.

Conclusion

R3CD demonstrates that by leveraging global awareness through transformers and fine-grained contrastive control over the diffusion process, AI can move beyond simple spatial placement to master complex, semantic interactions. Future work may extend this contrastive control to video generation or interactive scene editing.

Summary updated

Abstract

Image generation tasks have achieved remarkable performance using large-scale diffusion models. However, these models are limited to capturing the abstract relations (viz., interactions excluding positional relations) among multiple entities of complex scene graphs. Two main problems exist: 1) fail to depict more concise and accurate interactions via abstract relations; 2) fail to generate complete entities. To address that, we propose a novel Relation-aware Compositional Contrastive Control Diffusion method, dubbed as R3CD, that leverages large-scale diffusion models to learn abstract interactions from scene graphs. Herein, a scene graph transformer based on node and edge encoding is first designed to perceive both local and global information from input scene graphs, whose embeddings are initialized by a T5 model. Then a joint contrastive loss based on attention maps and denoising steps is developed to control the diffusion model to understand and further generate images, whose spatial structures and interaction features are consistent with a priori relation. Extensive experiments are conducted on two datasets: Visual Genome and COCO-Stuff, and demonstrate that the proposal outperforms existing models both in quantitative and qualitative metrics to generate more realistic and diverse images according to different scene graph specifications.
BibTeX
@inproceedings{liu2024r,
  author = {Jinxiu Liu and Qi Liu},
  title = {{R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=NvqJZebpRU},
}
Figure