LayoutRAG: Retrieval-Augmented Model for Content-agnostic Conditional Layout Generation

Yuxuan Wu, Le Wang, Sanping Zhou, Mengnan Liu, Gang Hua, Haoxiang LiarXiv 2025 · DOI · arXiv

Introduction

In graphic design and user interface (UI) development, layout generation involves the strategic arrangement of visual components like text, images, and buttons. Modern applications require controllable generation where users provide sparse constraints (e.g., "three text boxes and one image"), leaving the model to determine optimal sizing and positioning. However, current diffusion and flow-matching models often struggle with these under-specified conditions, frequently producing layouts with unnatural overlaps or excessive empty space. Furthermore, correcting these models typically requires expensive retraining.

To address these challenges, LayoutRAG introduces Retrieval-Augmented Generation (RAG) to the geometric domain. By retrieving high-quality templates from a database to guide the generative process, LayoutRAG improves structural integrity and allows for model updates via simple database swaps without retraining.

Overview of the LayoutRAG framework comparing standard conditional generation with the retrieval-augmented approach.

Methodology

The LayoutRAG framework decouples the generation process into two distinct stages: retrieval and augmented generation.

1. Layout Retrieval

Retrieving geometric layouts is complex due to the mix of discrete element types and continuous coordinates. LayoutRAG utilizes a category count-based index system. When a user specifies constraints, the system filters the database for layouts meeting the minimum category requirements. It then employs bipartite matching based on Intersection over Union (IoU) to identify the most structurally similar template from the database.

2. Retrieval-Augmented Generation

The generative core is built upon LayoutFlow, a flow-matching framework. To integrate the retrieved reference, the authors developed a Reference Network featuring a Condition Modulated Attention (CMA) module. Unlike simple concatenation, CMA dynamically fuses the current generation state, the reference layout, and user constraints. This ensures the model selectively adopts structural hints from the reference while strictly adhering to the user's specific requirements.

Detailed architecture of the Reference Network and the Condition-Modulated Attention (CMA) mechanism.

Key Insights

  • Geometric RAG: The paper demonstrates that RAG is highly effective for abstract geometric arrangements, providing a "cheat sheet" for implicit design rules like margins and visual hierarchy that are often missing from sparse user prompts.
  • Zero-Shot Adaptability: Model behavior can be updated or corrected by simply swapping the retrieval database. This allows engineers to adapt to new design trends or fix failure cases without costly fine-tuning.
  • Selective Fusion: The CMA module is critical; it prevents the model from blindly copying irrelevant parts of the retrieved template, maintaining a balance between reference guidance and user constraints.

Experimental Results

LayoutRAG was evaluated on the RICO (66k UI layouts) and PubLayNet (360k document layouts) datasets, filtered to layouts with 20 or fewer elements.

  • Quantitative Performance: LayoutRAG achieved state-of-the-art (SOTA) results across nearly all metrics. On the Rico dataset for the class-to-size+position (C->S+P) task, it achieved a Frechet Inception Distance (FID) of 1.22, significantly outperforming LayoutFlow (1.48) and LayoutDiffusion (1.56).
  • Qualitative Quality: Visual analysis confirms that LayoutRAG produces cleaner alignments and fewer overlaps than baseline models, more closely resembling human-designed layouts.
  • Ablation Studies: Replacing the CMA module with standard cross-attention led to a performance drop (FID increased from 0.77 to 0.90 on PubLayNet), validating the specialized attention mechanism.

Limitations and Future Work

The effectiveness of the retrieval mechanism is currently tied to the diversity of element categories; datasets with fewer categories (like PubLayNet) see smaller gains than those with many (like Rico). Additionally, in highly constrained completion tasks, the model may occasionally prioritize template adherence over the underlying data distribution. Future research will explore content-aware generation, incorporating actual text and image semantics into the RAG framework.

Summary updated

Abstract

Controllable layout generation aims to create plausible visual arrangements of element bounding boxes within a graphic design according to certain optional constraints, such as the type or position of a specific component. While recent diffusion or flow-matching models have achieved considerable advances in multifarious conditional generation tasks, there remains considerable room for generating optimal arrangements under given conditions. In this work, we propose to carry out layout generation through retrieving by conditions and reference-guided generation. Specifically, we retrieve appropriate layout templates according to given conditions as references. The references are then utilized to guide the denoising or flow-based transport process. By retrieving layouts compatible with the given conditions, we can uncover the potential information not explicitly provided in the given condition. Such an approach offers more effective guidance to the model during the generation process, in contrast to previous models that feed the condition to the model and let the model infer the unprovided layout attributes directly. Meanwhile, we design a condition-modulated attention that selectively absorbs retrieval knowledge, adapting to the difference between retrieved templates and given conditions. Extensive experiment results show that our method successfully produces high-quality layouts that meet the given conditions and outperforms existing state-of-the-art models. Code will be released upon acceptance.
BibTeX
@misc{wu2025layoutrag,
  author = {Yuxuan Wu and Le Wang and Sanping Zhou and Mengnan Liu and Gang Hua and Haoxiang Li},
  title = {{LayoutRAG: Retrieval-Augmented Model for Content-agnostic Conditional Layout Generation}},
  year = {2025},
  eprint = {2506.02697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02697},
}
Figure