Autoregressive Layout Synthesis

35 concepts28 papers1 proposal

When humans design a room, arrange a dinner table, or compose a graphic poster, we rarely place every element simultaneously. Instead, we work sequentially: we position the largest or most critical object first—like a bed or a primary image—and then arrange secondary items relative to it.

Yet, early generative models for scene synthesis attempted to generate entire layouts all at once. This simultaneous approach forces a network to navigate a massive, complex joint probability distribution. Without an explicit mechanism to coordinate spatial relationships, models frequently produce physical impossibilities: chairs clipping through tables, text overlapping logos, or vehicles colliding in simulated intersections.

To overcome this, a prominent research direction treats scene generation as a sequential decision-making process. By breaking down the joint probability distribution into a series of conditional steps, autoregressive layout synthesis allows models to build complex, physically coherent environments one object at a time.

The Chain Rule of Scene Generation

The core mathematical intuition behind this approach relies on the chain rule of probability. Instead of modeling the joint probability of all objects $P(O_1, O_2, \dots, O_N)$ simultaneously, the problem is factored into a product of conditional probabilities:

$$P(O_1, O_2, \dots, O_N) = \prod_{i=1}^N P(O_i \mid O_1, \dots, O_{i-1})$$

As noted by the authors of SceneGen, this autoregressive decomposition drastically simplifies multi-actor environments by transforming a single high-dimensional problem into a sequence of simpler, context-dependent placement steps.

When a model places a new object, it conditions its decision on the exact state of everything already established in the scene. In GENESIS, Martin Engelcke et al. demonstrated that an autoregressive prior over object-centric latents naturally learns a "scene grammar." The model inherently learns to avoid physical overlaps and establishes a natural hierarchy, such as rendering global background features before placing foreground objects. Similarly, Azimkhon Ostonov et al. in RLSS framed this as a reinforcement learning problem where an agent adds objects to a 3D environment one by one, explicitly evaluating the environment's current state before each addition to satisfy rigid spatial constraints.

Treating Layouts as Language

How do we parameterize a scene so a neural network can process it sequentially? A highly effective strategy is to discretize spatial coordinates and attributes into tokens, effectively translating scene synthesis into a language modeling task.

In SceneFormer, Xinpeng Wang et al. treated indoor layouts as a "sentence" where each word is a piece of furniture defined by its category, location, and orientation. Using a Transformer decoder, the model implicitly learns complex spatial dependencies across multiple planes—such as placing an air conditioner on a wall or a lamp on a ceiling—without requiring explicit rule-based constraints. This concept is extended to general visual priors in Learning Visual Prior via Generative Pre-Training, where the authors trained a transformer decoder to maximize the next-token likelihood of serialized bounding boxes and poses, capturing real-world co-occurrence patterns through standard next-token prediction.

To maintain structural coherence, models often employ a hierarchical generation strategy. For example, Yi-Fu Wu et al. introduced Neural Language of Thought Models, which first predict an object-level token and then sequentially generate its constituent attribute tokens (such as color and shape). This hierarchical construction ensures that the model focuses on object-by-object consistency before filling in fine-grained details.

Structuring the Action Space: "Where" and "What"

Autoregressive conditioning does not stop at selecting the next object category; it is also used to decompose the complex attributes within a single object. Predicting a 3D coordinate, scale, rotation, and shape simultaneously is a difficult high-dimensional leap.

To address this, Donghoon Lee 0004 et al. in Context-aware Synthesis and Placement of Object Instances split object insertion into two sequential modules: a placement module first determines the spatial coordinates and scale, and a shape module subsequently generates a category-specific mask. By conditioning the shape generation on a fixed local context determined by the placement module, the search space is significantly reduced.

A similar decomposition is utilized in 3D molecular design by Gregor N. C. Simm et al. in Symmetry-Aware Actor-Critic for 3D Molecular Design, where the task of placing atoms in 3D space is broken down into a hierarchy of dependent choices: a focal point, a component type, a distance, and an orientation.

Furthermore, layout and geometry can be tightly coupled in a single autoregressive pass. In RoomDesigner, Yiqun Zhao et al. predicted spatial parameters and geometric tokens simultaneously for each furniture item, capturing the intricate statistical dependencies between an object's style and its physical placement.

Universal Applicability Across Domains

The intuition of sequential layout synthesis is remarkably versatile, demonstrating success across vastly different domains:

  • Graphic Design and Presentation Layouts: In PosterLayout, HsiaoYuan Hsu et al. modeled graphic design as a sequential process where elements like logos, text, and decorations are placed in a prioritized order. This ensures subsequent elements respect the boundaries of earlier ones, mimicking human design logic. Similarly, Hsin-Ying Lee et al. developed the Neural Design Network, which iteratively generates layout coordinates one element at a time using teacher forcing to maintain spatial consistency across a canvas.
  • Traffic Scenario Generation: For autonomous driving simulation, Lan Feng developed TrafficGen, an autoregressive framework that sequentially places vehicles on a map. The scene context is dynamically updated after each vehicle is added, ensuring that new agents respect the spatial constraints and velocities of previously placed traffic.
  • Robotics and Compositional Media: In Imagine This!, Tanmay Gupta et al. generated multi-character video layouts by sequentially predicting bounding boxes, leveraging spatial context to ensure characters interact naturally with environment entities. In robotics, Weiyu Liu et al. introduced StructFormer, which uses an encoder-decoder transformer to predict 6-DoF pose offsets autoregressively, ensuring that the placement of each object respects the global context of a desired semantic structure (like a table setting).
  • 3D Parts and Web Layouts: Minghao Chen et al. in AutoPartGen applied this to 3D geometry, generating objects as sequences of discrete, semantically meaningful parts to prevent gaps and overlaps. In user interface design, Shuai Xiao et al. introduced Tile Networks, which treat whole-page recommendation layouts as a joint selection task, choosing an item and its 2D grid coordinate at each decoding step based on geometric relationships. Even semantic segmentation maps can be built this way; Yen-Chi Cheng et al. developed SegVAE to construct semantic maps category-by-category, allowing secondary features to conform to primary structural anchors.
  • Human-Conditioned Environments: Hongwei Yi et al. developed MIME, an autoregressive transformer that sequentially predicts 3D object attributes conditioned on human motion paths, ensuring generated furniture supports actions like sitting without obstructing walking paths.

Why the Feedback Loop Matters

The fundamental reason autoregressive layout synthesis succeeds where simultaneous generation fails lies in its dynamic feedback loop.

To understand why this is critical, consider the alternative. In BOOTPLACE, Hang Zhou et al. observed that a major limitation of non-sequential placement models is their inability to dynamically update scene context as new objects are introduced. When a model lacks a mechanism to ingest its own prior outputs, it cannot adapt to the shrinking physical space or the shifting semantic context of the canvas.

Autoregressive synthesis functions as a rolling constraint satisfaction mechanism. Every time an object is placed, the environment's state is updated, and the available "slots" change. By forcing the network to read the updated environment before writing the next element, the model naturally avoids spatial collisions and maintains global coherence, turning a chaotic combinatorial problem into a structured, context-aware progression.

Key Foundational Work

Imagine This! Scripts to Compositions to Videos (2018) — contributed 2 concept elements to this direction.
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018) — contributed 1 concept elements to this direction.
Context-aware Synthesis and Placement of Object Instances (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_object_placement_7afd8ab5,
  title = {sequential object placement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-object-placement/rdr_7afd8ab5909040a195caa0a5bd7a3ed3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_object_placement_7afd8ab5_v1,
  title = {sequential object placement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-object-placement/rdr_7afd8ab5909040a195caa0a5bd7a3ed3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake