Abstract
Automating graphic design requires a delicate balance between element categories, sizes, positions, and complex spatial relationships. While existing layout generation models often specialize in narrow tasks—such as generating designs from scratch or refining existing sketches—they frequently struggle with mixed-constraint scenarios. This paper discusses the Layout Diffusion Generative Model (LDGM), a unified framework that treats layout generation as a denoising process. By decoupling the noise-adding process for discrete and continuous attributes while denoising them jointly, LDGM achieves state-of-the-art performance across diverse design domains, including magazines, mobile UIs, and documents.
Introduction
Graphic design is fundamentally a structural problem. A successful layout is not merely a collection of items but a coherent arrangement where element categories (e.g., "image," "header," "button") and geometric properties (position and scale) must align with both aesthetic principles and functional constraints.
Existing AI models for layout generation typically fall into two categories: specialists and limited generalists. Specialists excel at specific tasks, such as generating a layout from a predefined list of element types. Generalists attempt broader applications but are often restricted to specific subtasks and struggle when presented with "messy" inputs—such as a mix of precise coordinates, rough sketches, and missing attributes. LDGM addresses these limitations by framing all layout tasks as a single objective: recovering a clean layout from a corrupted or incomplete input signal.
Methodology
LDGM leverages the principles of Diffusion Models, which generate data by learning to reverse a gradual corruption process. The framework introduces three primary innovations to adapt diffusion to the unique requirements of graphic design.
1. Decoupled Diffusion
Layout attributes are heterogeneous; a "category" is a discrete label, whereas "geometry" (x, y coordinates, width, and height) is spatial and continuous. LDGM employs a decoupled strategy to handle these differences:
- Categories: These use uniform noise, where labels are randomly swapped during the corruption phase.
- Geometry: These attributes use discretized Gaussian noise. Unlike uniform noise, this respects the spatial "distance" between coordinates, acknowledging that a small shift in position is a less severe corruption than a total relocation.
By allowing these attributes to follow their own corruption "timelines," the model is exposed to a more diverse range of training samples, improving its robustness.
2. Joint Denoising Architecture
While noise is added to attributes separately, the denoising process is performed jointly. LDGM utilizes a Transformer-based architecture that processes the global context of all elements and their relationships simultaneously. This ensures that the model accounts for inter-element dependencies—such as ensuring an icon is correctly placed relative to its associated text—during the reconstruction phase.
3. Confidence-based Inference
During the generation phase, the model does not attempt to predict all missing information in a single pass. Instead, it employs a confidence-based mechanism: it predicts attributes, retains those it is most "confident" about, and re-evaluates the remaining attributes in subsequent steps. This iterative refinement prevents minor errors in early stages from compounding into structural failures in the final layout.
Experimental Evaluation
The model was evaluated on three benchmark datasets representing distinct design challenges:
- Magazine: Focused on aesthetic and creative layouts (4K+ pages).
- Rico: Mobile UI layouts characterized by high element density and complex hierarchies (66K+ layouts).
- PubLayNet: Structured document images where alignment and formal hierarchy are critical (360K+ images).
Key Results
LDGM consistently outperformed previous state-of-the-art models, such as UniLayout and BLT, across several metrics:
- Realism: The model achieved significantly lower Fréchet Inception Distance (FID) scores, particularly on the Rico UI dataset, indicating that its generated layouts are statistically closer to human-designed ones.
- Aesthetics: LDGM showed superior performance in alignment and overlap metrics, producing neatly arranged elements that respect spatial boundaries.
- Versatility: The model successfully handled the "Gen-PCM" task—a complex scenario involving a mixture of precise, coarse, and missing attributes—demonstrating its capability as a true generalist.
Discussion and Limitations
LDGM represents a significant advancement in viewing layout generation as a corrupted signal recovery problem. By treating a user's partial input as a "fully corrupted" spatial signal but a "preserved" categorical signal, a single model can handle virtually any combination of constraints.
However, some limitations remain. The model currently requires a fixed element count ($N$) to be specified by the user beforehand. Additionally, the reliance on discrete quantization for coordinates, while standard, may limit the capture of extremely fine design nuances. Future research may explore continuous diffusion processes and the extension of this decoupled approach to other design attributes, such as color palettes, typography, and actual image content.
Summary updated