Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation

Mohammad Amin Shabani, Zhaowen Wang, Difan Liu, Nanxuan Zhao, Jimei Yang, Yasutaka Furukawa

Abstract

Design layout generation requires a delicate balance between visual aesthetics and structural editability. Traditional models often excel in one domain while failing in the other: vector-only models lack visual awareness, while image-only models produce uneditable flat rasters. Presented at CVPR 2024, the Visual Layout Composer (VLC) introduces a dual-domain diffusion approach. By synchronizing a latent image diffusion process with a vector coordinate diffusion process through shared attention maps, VLC generates layouts that are both visually intelligent and fully editable.

Introduction

Generating high-quality layouts, such as posters or magazine covers, involves more than simple box placement. Effective designs must respect "salient regions"—for instance, avoiding placing text over a subject's face. Existing AI design tools suffer from two primary gaps: visual blindness, where models treat elements as abstract categories without understanding content, and the "flat image" problem, where generative outputs lack the layer-based structure required for professional editing. VLC addresses these by merging the visual reasoning of large-scale image models with the structural precision of vector models.

Methods: The Dual Diffusion Architecture

The core of VLC is the Image-Vector Dual Diffusion Model, which runs two parallel diffusion processes:

  • The Image Branch: Utilizing a pre-trained Stable Diffusion (v1.5) model, this branch "renders" the design's appearance. While most layers remain frozen, cross-attention modules are trained to interpret input elements like images and text.
  • The Vector Branch: A transformer-based denoiser predicts 4D coordinates (x, y, width, height) for every layout element.
  • Attention Modulation: This "bridge" extracts spatial attention maps from the Image Branch and feeds them into the Vector Branch. This allows the vector model to "see" visual focal points, aligning bounding boxes with image features.
  • Canvas Masking: A specialized technique handles non-square aspect ratios by zeroing out latent features outside the intended canvas during denoising.
The VLC architecture synchronizes image and vector diffusion branches via attention map exchange.

Datasets and Evaluation

The authors evaluated VLC using standard benchmarks (Crello, RICO, PubLayNet) and introduced a massive new Poster Dataset containing 140,000 image-layout pairs—six times larger than previous visual layout resources. This scale allows the model to learn complex relationships between high-resolution imagery and professional typography placement.

Key Results and Insights

VLC achieved state-of-the-art (SOTA) performance, notably reaching a Bounding Box FID of 0.09 on the Poster dataset, a significant improvement over previous models like LayoutDM (1.81).

Beyond metrics, the model demonstrates superior visual-aware placement. By leveraging CLIP embeddings, it learns to avoid occluding important subjects and maintains color harmony. The dual-diffusion nature allows for iterative refinement; as the image branch suggests a visual composition, the vector branch adjusts coordinates in real-time to match. This enables advanced applications like layout style transfer and intelligent canvas resizing.

The final output is a composed vector design with fully editable layers.

Limitations and Future Work

Despite its success, VLC faces challenges regarding computational overhead due to running two parallel diffusion models. Additionally, the authors noted that standard metrics like FID may not fully capture "design logic," suggesting a need for more design-centric evaluation frameworks. Future work may focus on optimizing the dual-branch speed and improving robustness against noisy training data. VLC represents a major step toward AI tools that understand the fundamental principles of graphic design rather than just generating pixels.

Summary updated

Abstract

This paper proposes an image-vector dual diffusion model for generative layout design. Distinct from prior efforts that mostly ignores visual information of elements and the whole canvas, our approach integrates the power of a pre-trained large image diffusion model to guide layout composition in a vector diffusion model by providing enhanced salient region understanding and high-level inter-element relationship reasoning. Our proposed model simultaneously operates in two domains: it generates the overall design appearance in the image domain while optimizing the size and position of each design element in the vector domain. The proposed method achieves the state-of-the-art results on several datasets and enables new layout design applications. Project webpage: https://aminshabani.github.io/visual_layout_composer.
BibTeX
@inproceedings{shabani2024visual,
  author = {Mohammad Amin Shabani and Zhaowen Wang and Difan Liu and Nanxuan Zhao and Jimei Yang and Yasutaka Furukawa},
  title = {{Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00881},
}
Figure