Unifying Layout Generation with a Decoupled Diffusion Model

Mude Hui, Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yuwang Wang, Yan Lu

TL;DR

The Layout Diffusion Generative Model (LDGM) unifies diverse layout generation tasks—ranging from creating designs from scratch to refining messy drafts—into a single, versatile framework. By treating missing or imprecise layout attributes as "noisy" states in a decoupled diffusion process, LDGM achieves state-of-the-art performance across magazine, document, and user interface (UI) datasets.

What Problem Does This Paper Address?

Graphic layout generation is a complex balancing act. A designer (or an AI) must manage element categories, sizes, positions, and the relationships between them to create something both functional and aesthetic. Historically, researchers have treated different layout tasks as separate problems:

  • Unconditional Generation: Creating a layout from nothing.
  • Conditional Generation: Creating a layout based on specific constraints (e.g., "put an icon here" or "make this text larger").
  • Refinement: Taking a "coarse" or messy layout and cleaning it up.
  • Completion: Filling in missing pieces of a partial design.

Previous state-of-the-art models were often specialized for just one or two of these tasks. Even "unified" models like UniLayout relied on a limited set of predefined subtasks. The challenge lies in the fact that different layout attributes have different "semantics"—a category label is a discrete choice with no inherent distance between options, while a coordinate position is a spatial value where proximity matters deeply.

Methods

The authors propose LDGM, which reframes layout generation as a denoising process. The core innovation is the Decoupled Diffusion Model.

Standard diffusion models add noise to data until it becomes unrecognizable and then learn to reverse that process. LDGM applies this to layouts by viewing missing or coarse attributes as intermediate "noisy" statuses of a completed layout.

Figure 1. The layout generation tasks can be unified into a diffu- sion (noise-adding) process and a generation (denoising) process.

Key technical components include:

  1. Attribute-Specific Noise: Instead of a one-size-fits-all approach, LDGM uses different noise strategies. It uses uniform noise for discrete categories and discretized Gaussian noise for spatial attributes (position and size), allowing the model to understand spatial distances.
  2. Decoupled Timelines: During training, the model doesn't just add noise to everything at once. It uses separate timelines for categories, positions, and sizes. This "decoupling" ensures the model sees a massive variety of corrupted states, making it robust enough to handle any combination of user inputs at inference time.
  3. Transformer-based Denoising: The heart of the model is a transformer that performs a joint reverse process. It uses self-attention to pass messages across all elements and their attributes, exploiting global context to ensure the final layout is coherent.
  4. Confidence-based Inference: Rather than predicting everything in one go, the model uses a multi-step denoising strategy. For missing attributes, it predicts them and keeps only the most confident results, re-masking the rest for the next step.
Figure 3. Overall framework of our method. The input attributes of different subtasks can be considered as different xt. LDGM gradually denoise them to x0 as the final generation results.

Datasets

The model was evaluated on three standard benchmarks:

  • Magazine: 4K+ images of magazine pages with 6 element categories.
  • Rico: 66K+ mobile app UI layouts with 13 categories (cleaned version).
  • PubLayNet: 360K+ document layout examples with 5 categories.

Key Insights and Ideas

  • Layout as a Corrupted State: The "aha moment" is realizing that every layout task—whether it's completion, refinement, or generation—is just a different point on a diffusion timeline. A missing element is just "maximum noise," while a coarse element is "partial noise."
  • Semantics-Aware Denoising: By decoupling the diffusion process, the authors acknowledge that "where an element is" and "what an element is" are fundamentally different types of information that require different mathematical treatments.
  • Versatility through Randomization: By training on parallel, decoupled timelines, the model naturally learns to handle "combinational" tasks that were previously impossible, such as refining some elements while simultaneously generating others from scratch.

Key Results

LDGM consistently outperformed existing models like LayoutTrans, BLT, and UniLayout across all datasets and metrics (FID, MaxIoU, Alignment, and Overlap).

  • Quantitative: On the Rico dataset, LDGM achieved an FID (a measure of layout quality/realism) of 16.64, significantly beating UniLayout's 18.06 and BLT's 39.64.
  • Qualitative: Visual comparisons show that LDGM produces layouts with much better alignment and fewer overlapping elements compared to previous SOTA methods.
Figure 4. Qualitative comparisons with state-of-the-art layout generation methods.

The model also proved its "comprehensive versatility" by succeeding in newly proposed general tasks (like Gen-PCM, which mixes precise, coarse, and missing attributes) that previous models simply could not attempt.

Limitations and Future Work

While LDGM is highly versatile, the paper notes that the current approach relies on a fixed number of elements ($N$) provided by the user. Future work could involve predicting the number of elements automatically. Additionally, while the model handles spatial relations well, integrating more complex logical or semantic relations between elements remains an open area for exploration. As with most diffusion models, inference speed (requiring multiple denoising steps) may also be a consideration for real-time applications compared to simple one-pass autoregressive models.

Summary updated

Abstract

Layout generation aims to synthesize realistic graphic scenes consisting of elements with different attributes including category, size, position, and between-element relation. It is a crucial task for reducing the burden on heavy-duty graphic design works for formatted scenes, e.g., publications, documents, and user interfaces (UIs). Diverse application scenarios impose a big challenge in unifying various layout generation subtasks, including conditional and unconditional generation. In this paper, we propose a Layout Diffusion Generative Model (LDGM) to achieve such unification with a single decoupled diffusion model. LDGM views a layout of arbitrary missing or coarse element attributes as an intermediate diffusion status from a completed layout. Since different attributes have their individual semantics and characteristics, we propose to decouple the diffusion processes for them to improve the diversity of training samples and learn the reverse process jointly to exploit global-scope contexts for facilitating generation. As a result, our LDGM can generate layouts either from scratch or conditional on arbitrary available attributes. Extensive qualitative and quantitative experiments demonstrate our proposed LDGM outperforms existing layout generation models in both functionality and performance.
BibTeX
@inproceedings{hui2023unifying,
  author = {Mude Hui and Zhizheng Zhang and Xiaoyi Zhang and Wenxuan Xie and Yuwang Wang and Yan Lu},
  title = {{Unifying Layout Generation with a Decoupled Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00193},
}

Loading related content...

Figure