VarScene: A Deep Generative Model for Realistic Scene Graph Synthesis

Tathagat Verma, Abir De, Yateesh Agrawal, Vishwa Vinay, Soumen Chakrabarti

TL;DR

VARSCENE is a novel variational autoencoder (VAE) designed to synthesize realistic scene graphs—the structural blueprints of images. By representing graphs as collections of "star" substructures and optimizing for distribution similarity (Maximum Mean Discrepancy) rather than just reconstruction, it successfully handles the massive vocabularies and complex semantics of real-world visual scenes that stymie traditional graph generators.

What Problem Does This Paper Address?

Scene graphs represent the "who, what, and where" of an image by modeling objects as nodes and their relationships as edges (e.g., man $\rightarrow$ wearing $\rightarrow$ shirt). While we have become proficient at extracting these graphs from images, generating novel, realistic scene graphs from scratch remains a major hurdle.

Existing graph generative models were primarily built for chemistry. Molecular graphs are relatively simple: they have tiny vocabularies (usually <15 atom types) and follow strict, well-defined rules like valency. In contrast, a dataset like Visual Genome contains over 75,000 object categories and 40,000 relationship types. Furthermore, the "rules" of visual scenes are latent and messy—there is no physical law preventing a model from generating a "car flying over a building," but it isn't "realistic" in a standard dataset.

The authors aim to bridge this gap by creating a model that captures the underlying semantic distribution of real-world scenes.

Methods

The authors introduce VARSCENE, which departs from standard graph generation in two fundamental ways:

  1. Star-based Representation: Instead of adding one node or edge at a time, VARSCENE views a graph as a collection of "stars." A star consists of a central "hub" node and its immediate incident edges ("spokes"). This captures local semantic context (e.g., a "person" star might include spokes for "wearing," "holding," and "standing on") more effectively than isolated nodes.
  2. MMD Optimization: Standard VAEs are trained to maximize the Evidence Lower Bound (ELBO), which focuses on reconstructing specific training instances. VARSCENE goes a step further: after an initial base training, the decoder is fine-tuned to minimize the Maximum Mean Discrepancy (MMD) between the distribution of generated graphs and the distribution of the ground-truth training set. This ensures the entire population of generated graphs looks statistically similar to real data.

The architecture uses a Graph Neural Network (GNN) encoder to transform stars into latent representations based on their distance from a randomly chosen "pivot" star. The decoder then samples stars sequentially from a learned distribution to assemble a new graph.

The neural architecture of the VARSCENE encoder and decoder.

Datasets

The model was evaluated on three established benchmarks of varying complexity:

  • Visual Genome (VG): The largest and most complex, featuring massive vocabularies (over 16,000 object types and 8,400 relation types in the evaluated split).
  • Small-sized Visual Genome (SVG): A cleaned subset with 150 object and 50 relation categories.
  • Visual Relationship Detection (VRD): A smaller dataset with 100 objects and 70 relations.

Key Insights and Ideas

  • Distribution over Instances: The core "aha moment" is that for scene synthesis, matching the statistical properties of the world (via MMD) is more important than perfectly reconstructing a single training example. This allows the model to generate novel but plausible variations.
  • Semantic Masking: To prevent nonsensical triples, the decoder uses a MASK function. It only allows the model to connect two stars with a specific relationship if that triple (Subject-Predicate-Object) has been seen at least once in the training data.
  • Parameter Efficiency: Despite handling much larger vocabularies, VARSCENE is remarkably lean. It uses significantly fewer parameters than baselines like MolGAN or GraphRNN because its complexity is independent of the dataset size.

Key Results

VARSCENE consistently outperformed state-of-the-art baselines (including DeepGMG, GraphRNN, and SceneGen) across multiple metrics:

  • Graph Similarity: It achieved significantly higher scores in Star-Sim, Edge-Sim, and Node-Sim, which measure how well the generated distributions of substructures match the test set.
  • Image Synthesis Quality: To test "realism," the authors fed generated graphs into a pre-trained scene-graph-to-image model (sg2im). Images generated from VARSCENE graphs had much better Inception Scores (IS) and lower Fréchet Inception Distances (FID) compared to baselines, meaning the synthesized scenes were more recognizable and diverse.
  • Performance Comparison: VARSCENE (both unconditional and conditional variants) dominates the competition across benchmarks.
Performance comparison of different graph generative models across three datasets.

Limitations and Future Work

  • Spatial Agnosticism: VARSCENE focuses purely on the semantics (the graph structure). It does not currently generate spatial layouts (bounding boxes or coordinates), though it can be paired with existing models that do.
  • Common Sense vs. Data: The model learns what is "realistic" solely from the training data. The authors suggest that incorporating external knowledge bases (like Wikidata) could provide a "common sense" prior to help the model generalize beyond the specific biases of a single dataset.
  • Bias Propagation: The authors explicitly note that any social biases (e.g., gender or racial stereotypes) present in the training scene graphs will likely be propagated to the generated output, a critical area for future mitigation strategies.

Summary updated

Abstract

Scene graphs are powerful abstractions that capture relationships between objects in images by modeling objects as nodes and relationships as edges. Generation of realistic synthetic scene graphs h...
BibTeX
@inproceedings{verma2022varscene,
  author = {Tathagat Verma and Abir De and Yateesh Agrawal and Vishwa Vinay and Soumen Chakrabarti},
  title = {{VarScene: A Deep Generative Model for Realistic Scene Graph Synthesis}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=YpQC75ztpnb},
}
Figure