HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

Bo Cheng, Yuhang Ma, Liebucha Wu, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, Yuhui Yin

Abstract

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a Hierarchical Controllable (HiCo) diffusion model for layout-to-image generation, featuring object separable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi-branch structure to represent hierarchy and aggregate them in a fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned.

Introduction

While text-to-image (T2I) models like Stable Diffusion have achieved remarkable visual fidelity, they often fail to adhere to precise spatial constraints. A common failure mode is the inability to correctly place multiple objects as specified in a prompt—for instance, swapping positions or merging distinct entities into a single "hybrid" object. Layout-to-Image (L2I) generation aims to solve this by using bounding boxes as additional conditioning.

However, current L2I approaches face significant hurdles. Training-free methods frequently suffer from "object missing," where the model ignores certain specified regions. Training-based methods, such as GLIGEN, can lead to "semantic bleeding," where attributes from one object leak into another due to the model processing a crowded layout in a single pass. HiCo addresses these challenges by treating foreground objects and backgrounds as distinct, hierarchical layers.

Hierarchical layout control showing nested bounding boxes for a person and accessories.

Methodology: Hierarchical Modeling

The core innovation of HiCo is spatial disentanglement through hierarchy. Rather than processing the entire layout holistically, HiCo decomposes the scene into manageable components:

  1. HiCo Net (Multi-branch Structure): The architecture employs separate conditioning branches for the global background and each individual foreground object. These branches share weights, maintaining efficiency while allowing the model to focus on local semantics.
  2. Hierarchical Processing: Each branch independently processes its assigned region's caption and bounding box. This isolation prevents interference between neighboring objects, ensuring that features for a specific entity are developed without semantic leakage.
  3. Fuse Net: A non-parametric module aggregates features from all branches using spatial masks. This "stitches" the isolated object features back into the global scene before integration into the main UNet's upsampling blocks.
  4. Compatibility: HiCo is designed as a modular addition to existing diffusion backbones (e.g., SD1.5, SDXL) and is compatible with acceleration techniques like LoRA and Latent Consistency Models (LCM).

Datasets and Benchmarking

To rigorously evaluate HiCo, the authors utilized a mix of existing and new datasets:

  • Training: A fine-grained subset of GRIT-20M (1.2 million pairs) and a coarse-grained subset of COCO Stuff (COCO-75K).
  • HiCo-7K Benchmark: A new, high-quality benchmark derived from GRIT-20M. The authors employed Grounding-DINO for re-validation and CLIP for semantic filtering, followed by extensive manual cleaning to ensure challenging and accurate layout-caption pairs.

Results and Analysis

HiCo demonstrates superior performance over state-of-the-art models like GLIGEN and InstanceDiff across multiple metrics.

Diverse layout-to-image generation results demonstrating spatial accuracy across various domains.

Quantitative Performance

On the HiCo-7K benchmark, HiCo achieved a LocalIoU Score of 59.98, significantly outperforming GLIGEN (48.54) and InstanceDiff (51.58). This indicates a much higher fidelity to the provided bounding boxes. In terms of image quality, HiCo reached an FID of 14.24, the lowest among all tested models.

Human Preference and Efficiency

In a user study, HiCo was preferred in 78.08% of cases regarding object quantity, spatial location, and aesthetic quality, compared to 67.76% for GLIGEN. Furthermore, despite its multi-branch architecture, HiCo's parallel inference mode ensures it remains highly efficient, boasting the shortest inference time among compared models.

Conclusion and Future Work

HiCo successfully mitigates the "object missing" problem by forcing the model to account for every specified box through its hierarchical branch structure. While it excels in spatial accuracy, current limitations include a lack of explicit depth understanding for complex occlusions (e.g., "A is behind B"). Future work involves developing depth-aware datasets and improving the integration of multiple distinct LoRA-based concepts in crowded scenes.

Summary updated

Abstract

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a \textbf{Hi}erarchical \textbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCo_T2I.
BibTeX
@inproceedings{cheng2024hico,
  author = {Bo Cheng and Yuhang Ma and Liebucha Wu and Shanyuan Liu and Ao Ma and Xiaoyu Wu and Dawei Leng and Yuhui Yin},
  title = {{HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=i9QpRjUAhv},
}
Figure