HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

Bocheng, YuhangMa, wuliebucha, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, Yuhui Yin

Generating an image from a text prompt is an impressive feat, but precisely placing objects within that image—known as layout-to-image (L2I) generation—remains a significant challenge. Current state-of-the-art models often stumble when layouts become crowded or complex. To address this, researchers have introduced HiCo (Hierarchical Controllable), a novel diffusion model architecture that masters complex layout-to-image generation by giving every object its own independent processing "lane."

The Challenge of Layout-to-Image Generation

While existing diffusion models excel at overall image synthesis, they frequently fail when forced to adhere to strict spatial conditions (like bounding boxes) for multiple objects. Common "bad cases" in current L2I generation include:

  • Object Omission: The model simply ignores one or more requested objects, especially in crowded scenes.
  • Semantic Bleeding: Attributes of one object (like color or texture) leak into another adjacent object.
  • Inconsistency: Lighting, style, or perspective shifts unnaturally between different parts of the image.

Existing solutions generally fall into two camps: training-free methods that manipulate attention maps (which often miss objects entirely) and training-based methods that add new layers (which can lead to image distortion). HiCo aims to solve these issues by fundamentally rethinking how spatial conditions are injected into the diffusion process.

Example of HiCo's layout-to-image generation, accurately placing multiple subjects (woman, child, dog) within specified bounding boxes.

The HiCo Solution: Divide, Conquer, and Fuse

The core innovation of HiCo is spatial disentanglement through hierarchical modeling. Instead of feeding the entire layout into the model as a single, easily confused block of information, HiCo breaks the generation process down:

  1. HiCo Net (Multi-branch Structure): The model employs multiple parallel branches that share weights. Each branch is responsible for a specific region defined by a bounding box and its associated caption. One branch handles the global background, while others focus entirely on individual foreground objects.
  2. Weight Sharing: By sharing weights across these branches, the model remains computationally efficient while ensuring that the "knowledge" of how to render objects is consistent across the entire image.
  3. Fuse Net: This acts as the aggregator. It takes the intermediate features from all the independent branches and combines them using a mask-based approach. This ensures that the features for one object stay strictly within its designated bounding box and do not interfere with neighboring elements.
  4. Scalability: HiCo is designed as a "plug-and-play" enhancement. It is compatible with various Stable Diffusion backbones (like SD1.5 and SDXL) and supports modern acceleration plugins like LoRA and Latent Consistency Models (LCM) for rapid inference.

Datasets and the HiCo-7K Benchmark

To train and rigorously evaluate HiCo, the researchers utilized several key datasets, including GRIT-20M and COCO-Stuff for large-scale training.

Crucially, the team introduced a new benchmark: HiCo-7K. Recognizing a lack of high-quality evaluation sets for multi-objective layouts, they curated 7,000 manually cleaned images with an average of 3.78 objects per image. This provides a much more rigorous test for spatial controllability than previous standards. They also utilized the standard COCO-3K subset for benchmarking against existing literature.

Key Results and Performance

HiCo achieves state-of-the-art performance across both quantitative and qualitative metrics, proving that treating layout generation as a set of independent sub-problems prevents the neural network from getting "confused" by overlapping constraints.

  • Better Fidelity: On the challenging new HiCo-7K benchmark, HiCo achieved a FID (Fréchet Inception Distance) of 14.24, significantly outperforming competitors like GLIGEN (19.65) and CAG (19.37).
  • Superior Precision: In terms of Average Precision (AP) for object placement, HiCo outperformed existing methods on COCO-3K, reaching an AP of 36.60.
  • Human Preference: In subjective evaluations, HiCo scored highest in "Overall" quality (78.08) compared to the next best model, InstanceDiff (70.54).
  • Efficiency: Despite its multi-branch structure, HiCo's parallel processing mode allows it to generate images faster than many existing serial-processing models as the number of objects increases.
HiCo successfully generates diverse and stylistically distinct concepts side-by-side based on precise layout conditions.

Limitations and Future Directions

While HiCo represents a major step forward for controllable generation, the authors note a few remaining hurdles. Currently, the model lacks an explicit mechanism to handle complex occlusion ordering (e.g., determining exactly which object is in front when bounding boxes overlap heavily), relying instead on the global text prompt for cues. Additionally, while compatible with LoRAs, generating images with multiple distinct LoRA-based concepts can still occasionally lead to feature mixing.

Future work will focus on curating data specifically for occlusion ordering and refining the multi-branch interaction to better handle diverse stylistic concepts simultaneously.

Summary updated

Abstract

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a \textbf{Hi}erarchical \textbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCo_T2I.
BibTeX
@inproceedings{bocheng2024hico,
  author = {Bocheng and YuhangMa and wuliebucha and Shanyuan Liu and Ao Ma and Xiaoyu Wu and Dawei Leng and Yuhui Yin},
  title = {{HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=i9QpRjUAhv},
}
Figure