Unifying Layout Generation with a Decoupled Diffusion Model

Mude Hui, Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yuwang Wang, Yan Lu

Abstract

Automating graphic design requires a delicate balance between element categories, sizes, positions, and complex spatial relationships. While existing layout generation models often specialize in narrow tasks—such as generating designs from scratch or refining existing sketches—they frequently struggle with mixed-constraint scenarios. This paper discusses the Layout Diffusion Generative Model (LDGM), a unified framework that treats layout generation as a denoising process. By decoupling the noise-adding process for discrete and continuous attributes while denoising them jointly, LDGM achieves state-of-the-art performance across diverse design domains, including magazines, mobile UIs, and documents.

Introduction

Graphic design is fundamentally a structural problem. A successful layout is not merely a collection of items but a coherent arrangement where element categories (e.g., "image," "header," "button") and geometric properties (position and scale) must align with both aesthetic principles and functional constraints.

Existing AI models for layout generation typically fall into two categories: specialists and limited generalists. Specialists excel at specific tasks, such as generating a layout from a predefined list of element types. Generalists attempt broader applications but are often restricted to specific subtasks and struggle when presented with "messy" inputs—such as a mix of precise coordinates, rough sketches, and missing attributes. LDGM addresses these limitations by framing all layout tasks as a single objective: recovering a clean layout from a corrupted or incomplete input signal.

Methodology

LDGM leverages the principles of Diffusion Models, which generate data by learning to reverse a gradual corruption process. The framework introduces three primary innovations to adapt diffusion to the unique requirements of graphic design.

1. Decoupled Diffusion

Layout attributes are heterogeneous; a "category" is a discrete label, whereas "geometry" (x, y coordinates, width, and height) is spatial and continuous. LDGM employs a decoupled strategy to handle these differences:

  • Categories: These use uniform noise, where labels are randomly swapped during the corruption phase.
  • Geometry: These attributes use discretized Gaussian noise. Unlike uniform noise, this respects the spatial "distance" between coordinates, acknowledging that a small shift in position is a less severe corruption than a total relocation.

By allowing these attributes to follow their own corruption "timelines," the model is exposed to a more diverse range of training samples, improving its robustness.

2. Joint Denoising Architecture

While noise is added to attributes separately, the denoising process is performed jointly. LDGM utilizes a Transformer-based architecture that processes the global context of all elements and their relationships simultaneously. This ensures that the model accounts for inter-element dependencies—such as ensuring an icon is correctly placed relative to its associated text—during the reconstruction phase.

3. Confidence-based Inference

During the generation phase, the model does not attempt to predict all missing information in a single pass. Instead, it employs a confidence-based mechanism: it predicts attributes, retains those it is most "confident" about, and re-evaluates the remaining attributes in subsequent steps. This iterative refinement prevents minor errors in early stages from compounding into structural failures in the final layout.

Experimental Evaluation

The model was evaluated on three benchmark datasets representing distinct design challenges:

  • Magazine: Focused on aesthetic and creative layouts (4K+ pages).
  • Rico: Mobile UI layouts characterized by high element density and complex hierarchies (66K+ layouts).
  • PubLayNet: Structured document images where alignment and formal hierarchy are critical (360K+ images).

Key Results

LDGM consistently outperformed previous state-of-the-art models, such as UniLayout and BLT, across several metrics:

  • Realism: The model achieved significantly lower Fréchet Inception Distance (FID) scores, particularly on the Rico UI dataset, indicating that its generated layouts are statistically closer to human-designed ones.
  • Aesthetics: LDGM showed superior performance in alignment and overlap metrics, producing neatly arranged elements that respect spatial boundaries.
  • Versatility: The model successfully handled the "Gen-PCM" task—a complex scenario involving a mixture of precise, coarse, and missing attributes—demonstrating its capability as a true generalist.

Discussion and Limitations

LDGM represents a significant advancement in viewing layout generation as a corrupted signal recovery problem. By treating a user's partial input as a "fully corrupted" spatial signal but a "preserved" categorical signal, a single model can handle virtually any combination of constraints.

However, some limitations remain. The model currently requires a fixed element count ($N$) to be specified by the user beforehand. Additionally, the reliance on discrete quantization for coordinates, while standard, may limit the capture of extremely fine design nuances. Future research may explore continuous diffusion processes and the extension of this decoupled approach to other design attributes, such as color palettes, typography, and actual image content.

Summary updated

Abstract

Layout generation aims to synthesize realistic graphic scenes consisting of elements with different attributes including category, size, position, and between-element relation. It is a crucial task for reducing the burden on heavy-duty graphic design works for formatted scenes, e.g., publications, documents, and user interfaces (UIs). Diverse application scenarios impose a big challenge in unifying various layout generation subtasks, including conditional and unconditional generation. In this paper, we propose a Layout Diffusion Generative Model (LDGM) to achieve such unification with a single decoupled diffusion model. LDGM views a layout of arbitrary missing or coarse element attributes as an intermediate diffusion status from a completed layout. Since different attributes have their individual semantics and characteristics, we propose to decouple the diffusion processes for them to improve the diversity of training samples and learn the reverse process jointly to exploit global-scope contexts for facilitating generation. As a result, our LDGM can generate layouts either from scratch or conditional on arbitrary available attributes. Extensive qualitative and quantitative experiments demonstrate our proposed LDGM outperforms existing layout generation models in both functionality and performance.
BibTeX
@inproceedings{hui2023unifying,
  author = {Mude Hui and Zhizheng Zhang and Xiaoyi Zhang and Wenxuan Xie and Yuwang Wang and Yan Lu},
  title = {{Unifying Layout Generation with a Decoupled Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00193},
}
Figure