STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generation

Ruyu Wang, Xuefeng Hou, Sabrina Schmedding, Marco HuberWACV 2025

TL;DR

This paper introduces STAY Diffusion, a novel diffusion-based model for generating realistic images from layout specifications (bounding boxes and labels). It tackles the challenge of controlling object appearance and relationships by introducing Edge-Aware Normalization and Styled-Mask Attention, achieving state-of-the-art results in generation diversity, accuracy, and controllability.

What Problem Does This Paper Address?

Generating realistic images from coarse layout information (like bounding boxes) is a challenging but valuable task for applications like image editing and data augmentation. Existing methods often face a trade-off between user flexibility and model control. While semantic maps offer strong guidance, they are difficult to create and modify. Layouts, on the other hand, are easier to manipulate but provide less precise information. Previous layout-to-image (L2I) synthesis methods, including GAN-based and early diffusion models, have struggled with issues like mode collapse, object interaction, and fine-grained control over object appearance. This paper aims to bridge this gap by developing a diffusion model that leverages layouts for precise control over object placement, appearance, and relationships, leading to diverse and accurate image generation.

Methods

STAY Diffusion builds upon the denoising diffusion probabilistic models (DDPM) framework. The core innovation lies in how it processes layout information and integrates it into the diffusion process.

  • Global Conditions: Layouts are transformed into "global conditions." This involves learning object representations that combine a category-specific embedding with a style-specific latent code ($z_{sty}$). A mask prediction subnetwork then generates initial object masks based on these representations.
  • Edge-Aware Normalization (EA Norm): This novel normalization layer is designed to fuse multiple objects smoothly and enable spatially-adaptive weight modulation. It utilizes self-supervised object masks predicted from layouts. To address overlapping mask issues, it constructs a "pixel-level weighted semantic map" by combining a non-overlapping map (prioritizing smaller objects) and an edge-aware map (extending object borders). This weighted map, along with object representations, is used to modulate normalization layer weights and biases.
  • Styled-Mask Attention (SM Attention): This module replaces the standard self-attention in the diffusion model's decoder. It uses a cross-attention mechanism that specifically attends to localized object regions based on the layout. It constructs queries from a semantic map derived from object masks and their representations, allowing the model to focus on relevant areas and better capture object relationships.

Datasets

The paper evaluates STAY Diffusion on two standard L2I synthesis benchmarks:

  • COCO-Stuff: This dataset contains annotations for both "thing" and "stuff" classes. The authors filtered it to include 3-8 objects per image, resulting in 171 categories.
  • Visual Genome (VG): This dataset provides dense annotations for objects, attributes, and relationships. The authors selected images with 3-30 bounding boxes.

Key Insights and Ideas

  • Layouts as a Sweet Spot: The paper argues that layouts strike a good balance between user flexibility and model control, offering a more practical input than complex semantic maps.
  • Self-Supervised Masks for Guidance: Instead of relying on pre-defined semantic maps, the model learns to generate its own "self-supervised maps" from layouts. This allows for greater adaptability, especially in domains where perfect semantic annotations are scarce.
  • The Importance of Mask Clarity: The authors hypothesize and demonstrate that the quality and clarity of these self-supervised masks significantly impact image fidelity and controllability. Imperfect masks can hinder generation quality.
  • Style Control via Latent Codes: The introduction of the $z_{sty}$ latent code allows for fine-grained control over object appearance, enabling users to resample this code to alter an object's style while keeping the layout and other conditions the same. This is a key differentiator from previous methods.

Key Results

  • State-of-the-Art Performance: STAY Diffusion outperforms previous state-of-the-art methods, including LayoutDM, on COCO-Stuff and VG datasets across several metrics.
  • Diversity, Accuracy, and Controllability: The model achieves superior performance in Diversity Score (DS), Classification Score (CAS), and YoloScore, indicating more varied, accurate, and recognizable object generation.
  • Competitive FID/IS: While excelling in other metrics, STAY Diffusion maintains competitive Fréchet Inception Distance (FID) and Inception Score (IS), suggesting high overall image quality.
  • Ablation Studies: The EA Norm and SM Attention modules are shown to be crucial. The full model (combining both) significantly outperforms baselines that use only one or simpler alternatives (like ISLA Norm). The EA Norm, in particular, leads to more integrated predicted masks.

Limitations and Future Work

  • Mask Quality Impact: The paper acknowledges that the quality of the self-supervised masks can affect image fidelity. While their method improves mask clarity, further improvements in mask generation could lead to even better results. They suggest that improving mask clarity is a promising direction for future work.
  • Domain Adaptability: While the authors claim their approach is more adaptable to domains beyond natural images (e.g., medical, industrial) due to not relying on pretrained LTGMs, this aspect is not extensively demonstrated in the experiments.
  • FID vs. Controllability Trade-off: The authors note that their tighter constraints (due to reliance on self-supervised maps) might lead to slightly lower FID compared to some methods, but this comes at the benefit of superior controllability.

Summary updated

BibTeX
@inproceedings{wang2025stay,
  author = {Ruyu Wang and Xuefeng Hou and Sabrina Schmedding and Marco Huber},
  title = {{STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure