TL;DR
Researchers have developed Salad, a dual-diffusion system that mimics human imagination by "hallucinating" complex scene details from simple, abstract text prompts. By evolving a basic scene graph into a detailed structural map using discrete diffusion, the model generates intricate, high-quality images without requiring users to write exhaustive, paragraph-long descriptions.
The "Prompting Tax" and the Abstract-to-Intricate Challenge
Current State-of-the-Art (SoTA) text-to-image (T2I) models like Stable Diffusion or DALL-E are incredibly powerful, but they come with a "prompting tax." To generate a high-quality, detailed image, users often must provide lengthy, hyper-specific descriptions. If you simply type "office," the model might produce a generic, empty room.
The authors identify this as the abstract-to-intricate challenge. There is a natural modality asymmetry between language and vision: a single word like "kitchen" implies a vast array of objects (stoves, tiles, fruit bowls) and relationships that aren't explicitly stated. Previous attempts to fix this involved using Large Language Models (LLMs) to expand text prompts. However, this often leads to visual distraction (the model focuses on the wrong new details) or wrong binding (attributes like "red" getting attached to the wrong object).
The Salad Framework: Hallucinating in Graph Space
To overcome these linguistic traps, the authors propose a framework called Salad (an SG-based hallucination diffusion system). Salad breaks the generation process into two distinct stages:
- Scene Graph Hallucination (SGH): Instead of expanding raw text, Salad converts the prompt into a Scene Graph (SG)—a structured representation of objects, their attributes, and their relationships. It uses a discrete diffusion model to iteratively "hallucinate" new nodes and edges, filling in the blanks of an abstract scene with plausible details (e.g., adding a "keyboard" and "monitor" to an "office" graph).
- Scene-driven Image Synthesis (SIS): The hallucinated graph then guides a standard Latent Diffusion Model (LDM). To ensure the image actually follows the graph, the authors introduce Hierarchical Scene Integration (HSI). This technique fuses graph data into the image generation process at four levels: individual objects, relational triplets, local regions, and the global scene.
To keep the results diverse, a Scene Sampling mechanism is used during inference, allowing the model to imagine many different versions of the same abstract prompt.
Datasets and Benchmarks
The model was trained and evaluated using standard benchmarks alongside a newly introduced subset:
- MS-COCO: The standard benchmark for T2I.
- Visual Genome (VG): Used to provide rich scene graph annotations for training the SGH module.
- COCO-A2I (New): Introduced specifically to test the "abstract-to-intricate" capability. It contains 2,005 pairs where the text is intentionally short and abstract, while the corresponding images are complex.
Key Insights and Results
By hallucinating in the "graph space" rather than the "text space," the model avoids the grammar and long-range dependency issues that confuse standard T2I models. SGH acts as a structured "mental model" that fills in the context before pixels are even drawn.
Salad achieved new SoTA results on the COCO dataset, but its real strength showed in the abstract-to-intricate tests:
- FID Improvement: On the COCO-A2I benchmark, Salad achieved an FID of 31.25, significantly outperforming the previous best model, Frido (40.36).
- Semantic Accuracy: Using the CLIP score and Triplet Recall (measuring how well the final image matches the intended scene graph), Salad consistently beat models that relied on LLM-based text expansion.
- Qualitative Superiority: Visual results show that while other models might ignore parts of an abstract prompt or create "messy" compositions when prompts are long, Salad produces crisp, logically organized scenes that feel "complete."
Limitations and Future Work
While powerful, the system has a few limitations. Its performance is partially capped by the quality of existing scene graph parsers used during training. Additionally, running two diffusion processes (one for the graph, one for the image) is more computationally expensive than a single-pass T2I model. Finally, the authors note a scarcity of dedicated datasets for abstract-to-intricate synthesis, requiring them to bootstrap their own.
Future directions include exploring even more complex graph structures or moving toward "unconditional" scene hallucination, where the model can dream up entire, logically consistent worlds from scratch.
Summary updated