Imagine That! Abstract-to-Intricate Text-to-Image Synthesis with Scene Graph Hallucination Diffusion

Shengqiong Wu, Hao Fei, Hanwang Zhang, Tat-Seng ChuaNeurIPS 2023 · OpenReview

TL;DR

Researchers have developed Salad, a dual-diffusion system that mimics human imagination by "hallucinating" complex scene details from simple, abstract text prompts. By evolving a basic scene graph into a detailed structural map using discrete diffusion, the model generates intricate, high-quality images without requiring users to write exhaustive, paragraph-long descriptions.

The "Prompting Tax" and the Abstract-to-Intricate Challenge

Current State-of-the-Art (SoTA) text-to-image (T2I) models like Stable Diffusion or DALL-E are incredibly powerful, but they come with a "prompting tax." To generate a high-quality, detailed image, users often must provide lengthy, hyper-specific descriptions. If you simply type "office," the model might produce a generic, empty room.

The authors identify this as the abstract-to-intricate challenge. There is a natural modality asymmetry between language and vision: a single word like "kitchen" implies a vast array of objects (stoves, tiles, fruit bowls) and relationships that aren't explicitly stated. Previous attempts to fix this involved using Large Language Models (LLMs) to expand text prompts. However, this often leads to visual distraction (the model focuses on the wrong new details) or wrong binding (attributes like "red" getting attached to the wrong object).

Comparison of text-based prompt enrichment vs. Scene Graph hallucination.

The Salad Framework: Hallucinating in Graph Space

To overcome these linguistic traps, the authors propose a framework called Salad (an SG-based hallucination diffusion system). Salad breaks the generation process into two distinct stages:

The Salad architecture featuring Scene Graph Hallucination and Scene-driven Image Synthesis.
  1. Scene Graph Hallucination (SGH): Instead of expanding raw text, Salad converts the prompt into a Scene Graph (SG)—a structured representation of objects, their attributes, and their relationships. It uses a discrete diffusion model to iteratively "hallucinate" new nodes and edges, filling in the blanks of an abstract scene with plausible details (e.g., adding a "keyboard" and "monitor" to an "office" graph).
  2. Scene-driven Image Synthesis (SIS): The hallucinated graph then guides a standard Latent Diffusion Model (LDM). To ensure the image actually follows the graph, the authors introduce Hierarchical Scene Integration (HSI). This technique fuses graph data into the image generation process at four levels: individual objects, relational triplets, local regions, and the global scene.

To keep the results diverse, a Scene Sampling mechanism is used during inference, allowing the model to imagine many different versions of the same abstract prompt.

Datasets and Benchmarks

The model was trained and evaluated using standard benchmarks alongside a newly introduced subset:

  • MS-COCO: The standard benchmark for T2I.
  • Visual Genome (VG): Used to provide rich scene graph annotations for training the SGH module.
  • COCO-A2I (New): Introduced specifically to test the "abstract-to-intricate" capability. It contains 2,005 pairs where the text is intentionally short and abstract, while the corresponding images are complex.

Key Insights and Results

By hallucinating in the "graph space" rather than the "text space," the model avoids the grammar and long-range dependency issues that confuse standard T2I models. SGH acts as a structured "mental model" that fills in the context before pixels are even drawn.

Salad achieved new SoTA results on the COCO dataset, but its real strength showed in the abstract-to-intricate tests:

  • FID Improvement: On the COCO-A2I benchmark, Salad achieved an FID of 31.25, significantly outperforming the previous best model, Frido (40.36).
  • Semantic Accuracy: Using the CLIP score and Triplet Recall (measuring how well the final image matches the intended scene graph), Salad consistently beat models that relied on LLM-based text expansion.
  • Qualitative Superiority: Visual results show that while other models might ignore parts of an abstract prompt or create "messy" compositions when prompts are long, Salad produces crisp, logically organized scenes that feel "complete."

Limitations and Future Work

While powerful, the system has a few limitations. Its performance is partially capped by the quality of existing scene graph parsers used during training. Additionally, running two diffusion processes (one for the graph, one for the image) is more computationally expensive than a single-pass T2I model. Finally, the authors note a scarcity of dedicated datasets for abstract-to-intricate synthesis, requiring them to bootstrap their own.

Future directions include exploring even more complex graph structures or moving toward "unconditional" scene hallucination, where the model can dream up entire, logically consistent worlds from scratch.

Summary updated

Abstract

In this work, we investigate the task of text-to-image (T2I) synthesis under the abstract-to-intricate setting, i.e., generating intricate visual content from simple abstract text prompts. Inspired by human imagination intuition, we propose a novel scene-graph hallucination (SGH) mechanism for effective abstract-to-intricate T2I synthesis. SGH carries out scene hallucination by expanding the initial scene graph (SG) of the input prompt with more feasible specific scene structures, in which the structured semantic representation of SG ensures high controllability of the intrinsic scene imagination. To approach the T2I synthesis, we deliberately build an SG-based hallucination diffusion system. First, we implement the SGH module based on the discrete diffusion technique, which evolves the SG structure by iteratively adding new scene elements. Then, we utilize another continuous-state diffusion model as the T2I synthesizer, where the overt image-generating process is navigated by the underlying semantic scene structure induced from the SGH module. On the benchmark COCO dataset, our system outperforms the existing best-performing T2I model by a significant margin, especially improving on the abstract-to-intricate T2I generation. Further in-depth analyses reveal how our methods advance.
BibTeX
@inproceedings{wu2023imagine,
  author = {Shengqiong Wu and Hao Fei and Hanwang Zhang and Tat-Seng Chua},
  title = {{Imagine That! Abstract-to-Intricate Text-to-Image Synthesis with Scene Graph Hallucination Diffusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=hSTaTBIUCj},
}
Figure