Scene Graph to Image Synthesis via Knowledge Consensus

Yang Wu, Pengxu Wei, Liang Lin

Introduction

Generating realistic images from scene graphs—structured representations of a scene (e.g., "sheep" behind "dog")—is a powerful way to control AI creativity. However, it is notoriously difficult. Most existing methods rely on auxiliary information like manually drawn bounding boxes or pixel-level masks to bridge the gap between abstract logic and pixels. This reliance leads to issues with data scarcity and semantic drift, where models ignore the relationships in the graph.

The authors propose the Knowledge Consensus Generative Model (KCGM), a causal inference-based framework that enables high-quality image synthesis using only scene graphs, without requiring layout or mask supervision. By aligning the "knowledge" extracted from real images, synthetic images, and latent representations, the model achieves superior semantic consistency and controllable generation.

Methodology

KCGM treats image generation as a causal system and consists of three main pillars: Graphic Information Disentanglement (GID), a Structured Knowledge Encoder (SKE), and Knowledge Consensus Optimization.

Comparison of traditional layout-based generation (a) and the proposed causal dependence framework (b).

The core innovation lies in Knowledge Consensus Optimization. The model tries to reach a "consensus" by minimizing the divergence between three different graph distributions: the ground-truth graph ($K$), the graph estimated from the real image ($K_X$), and the graph estimated from the generated image ($K_{\hat{X}}$). This forces the model to learn the spatial and relational logic of the world directly from the graph's structure.

Causal diagrams illustrating the progression toward the Knowledge Consensus model (c).

By treating causality as a regularizer, the authors ensure the image is a "logical effect" of the graph. Disentanglement is also key; by splitting the latent space into object-specific and relationship-specific zones, the model avoids "tangled" representations where changing one object accidentally alters another. This approach allows for implicit layout learning, where the pressure to reach a consensus forces the model to learn spatial arrangements without explicit bounding box supervision.

Experiments and Results

The model was evaluated on the Visual Genome (VG) dataset, a standard benchmark for scene-to-image tasks. KCGM significantly outperformed baselines that do not use layouts, achieving an Inception Score (IS) of 9.42 and a lower FID of 51.27. With additional standard enhancements like attention mechanisms, it reached an IS of 11.63 and FID of 27.46, rivaling models that use extra layout supervision.

Qualitatively, the model excels at "counterfactual" generation. For example, one can take a scene graph of a giraffe in a field, swap "giraffe" for "elephant," and the model generates a plausible elephant while maintaining the rest of the scene's context.

Conclusion

While KCGM shows impressive results at $128 \times 128$ resolution, scaling to higher resolutions and handling nonsensical input graphs remain challenges. Future work may explore deeper graph processing to further enhance performance, as results showed that adding more GCN layers improved performance.

Summary updated

Abstract

In this paper, we study graph-to-image generation conditioned exclusively on scene graphs, in which we seek to disentangle the veiled semantics between knowledge graphs and images. While most existing research resorts to laborious auxiliary information such as object layouts or segmentation masks, it is also of interest to unveil the generality of the model with limited supervision, moreover, avoiding extra cross-modal alignments. To tackle this challenge, we delve into the causality of the adversarial generation process, and reason out a new principle to realize a simultaneous semantic disentanglement with an alignment on target and model distributions. This principle is named knowledge consensus, which explicitly describes a triangle causal dependency among observed images, graph semantics and hidden visual representations. The consensus also determines a new graph-to-image generation framework, carried on several adversarial optimization objectives. Extensive experimental results demonstrate that, even conditioned only on scene graphs, our model surprisingly achieves superior performance on semantics-aware image generation, without losing the competence on manipulating the generation through knowledge graphs.
BibTeX
@inproceedings{wu2023scene,
  author = {Yang Wu and Pengxu Wei and Liang Lin},
  title = {{Scene Graph to Image Synthesis via Knowledge Consensus}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25387},
}
Figure