Decoupling Spatial Planning from Generative Synthesis

519 concepts307 papers3 proposals

Generative models like Stable Diffusion and DALL-E 3 are remarkably adept at rendering textures and styles, yet they frequently stumble on basic spatial logic. A prompt asking for "a cat to the left of a dog" might result in the opposite arrangement, or a request for "four apples" might yield five. These failures stem from a fundamental limitation: end-to-end diffusion models attempt to solve semantic reasoning and pixel synthesis simultaneously, often sacrificing the former for the latter.

A growing body of research suggests a more robust alternative: using Large Language Models (LLMs) as symbolic architects. By decoupling the high-level planning of "what goes where" from the low-level rendering of pixels or geometry, researchers are enabling generative systems to achieve unprecedented spatial precision.

The LLM as a Spatial Architect

The core intuition behind this direction is that LLMs, despite being trained primarily on text, possess a latent "world model" capable of sophisticated spatial reasoning. When a complex scene is decomposed into a structured plan—such as a list of bounding boxes or a set of geometric constraints—the generative process becomes much more manageable.

Leigang Qu et al. demonstrate this in LayoutLLM-T2I, where an LLM acts as a high-level planner that generates bounding box coordinates before any pixels are drawn. This approach leverages the model's internal reasoning to define object placement and composition, effectively serving as a structural scaffold for the diffusion model. Similarly, Long Lian et al. propose LLM-grounded Video Diffusion (LVD) in /paper/art_4f0c377d060f4e72aad569999c7abf6d, using LLMs to generate spatiotemporal plans (dynamic bounding boxes) that guide a video generator’s attention maps over time.

Space as Structured Code

Why are LLMs so good at proposing layouts? One key insight is their extensive training on structured data like code. Researchers have found that representing spatial relationships as code—rather than raw numerical lists—taps into the models' existing proficiency with logic and hierarchy.

In LayoutGPT, Weixi Feng et al. represent spatial coordinates and object dimensions as CSS code. This allows the LLM to use web development structures to reason about visual layouts, transforming a geometric problem into a code generation task. Taking this further, Yunzhi Zhang et al. introduce The Scene Language in /paper/art_ec073a69614d4145bd656e56b1a2c882, which translates natural language into executable programs. These programs define complex spatial relationships and object hierarchies, such as loops for repetitive elements, providing higher precision than traditional scene graphs.

From 2D Layouts to 3D Environments

This planning paradigm extends naturally into 3D space and robotics, where the cost of spatial errors is even higher. By serializing 3D geometric data into text-based coordinates, LLMs can interpret the state of a physical object and generate logical sub-goals.

For instance, Jianmeng Liu et al. developed VP-LLM (/paper/art_bca2d08f968747da9531844877fbad80), which breaks 3D volumes into "patches" treated as tokens. This allows the LLM to interpret detailed captions to reconstruct missing geometry with higher precision than diffusion-only methods. In the realm of robotics, Lirui Wang et al. show in GenSim that LLMs can automatically design and code robotic simulation tasks, translating abstract concepts into functional code that maintains physical consistency without manual coordinate mapping.

Iterative Refinement and Feedback Loops

Perhaps the most powerful aspect of using LLMs for layout planning is their ability to engage in self-correction. Because the layout is a symbolic intermediate representation, the system can "look" at its plan (or a draft of the generated image) and refine it.

The SLD (Self-correcting LLM-controlled Diffusion) framework by Tsung-Han Wu et al. (/paper/art_da68ed3016e4435ebf982a041fd9e739) turns generation into an iterative conversation. The LLM generates a layout, checks it against the prompt, and adjusts the bounding boxes to fix errors in numeracy or attribute binding. Similarly, Naoto Inoue et al. use Vision-Language Models in Type-R (/paper/art_77586ee1f082474b9e277b7cc76d4a7c) to act as layout planners for retouching typos, ensuring that corrected text integrates naturally with the existing visual design harmony.

By treating spatial arrangement as a reasoning problem rather than a pixel-prediction problem, these methods allow generative AI to move beyond "hallucinated" layouts toward structured, physically plausible, and semantically accurate scenes.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 316 papers
  2. Version 1 · 2026-10-04 · 362 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_llm_based_layout_planning_835ccb9c,
  title = {LLM-based layout planning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-based-layout-planning/rdr_835ccb9cf0044eee85cd7f00765ea480},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_llm_based_layout_planning_835ccb9c_v2,
  title = {LLM-based layout planning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/llm-based-layout-planning/rdr_835ccb9cf0044eee85cd7f00765ea480/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake