TL;DR
HiCo (Hierarchical Controllable) is a new diffusion model architecture that masters complex layout-to-image generation by treating foreground objects and backgrounds as distinct, hierarchical layers. By using a multi-branch conditioning structure with weight sharing, it effectively eliminates the common "missing object" problem and achieves state-of-the-art spatial accuracy and image quality.
What Problem Does This Paper Address?
While text-to-image models like Stable Diffusion are impressive, they are notoriously bad at following precise spatial instructions. If you ask for "a cat on the left and a dog on the right," you might get two cats, a "cat-dog" hybrid, or objects in the wrong places.
Existing "Layout-to-Image" (L2I) methods try to fix this using bounding boxes, but they hit a wall in complex scenes. Training-free methods often suffer from "object missing" (the model simply ignores some boxes), while training-based methods like GLIGEN can cause visual distortions or "semantic bleeding," where the attributes of one object accidentally leak into another because the model tries to process the entire crowded layout in a single pass.
Methods
The core innovation of HiCo is spatial disentanglement through hierarchy. Instead of feeding the entire layout into the model at once, HiCo breaks it down:
- HiCo Net (Multi-branch Structure): The model uses separate conditioning branches for the global background and each individual foreground object. These branches share weights (similar to the ControlNet philosophy) to remain efficient while focusing on local semantics.
- Hierarchical Modeling: Each branch independently processes its assigned region's caption and bounding box. This ensures that the "features" for a specific object are developed in isolation, preventing interference from neighboring objects.
- Fuse Net: This is a non-parametric module that aggregates the features from all branches using masks. It essentially "stitches" the isolated object features back into the global scene before they are integrated into the main UNet's upsampling blocks.
- Scalability: Because HiCo acts as a side-car to the main diffusion model, it is compatible with various backbones (SD1.5, SDXL) and acceleration plugins like LoRA and LCM (Latent Consistency Models).
Datasets
The authors utilized several datasets for a comprehensive evaluation:
- Training: A fine-grained subset of GRIT-20M (1.2 million pairs) and a coarse-grained subset of COCO Stuff (labeled as COCO-75K).
- Evaluation: The standard COCO-3K for benchmarking against existing models.
- New Benchmark (HiCo-7K): The authors introduced HiCo-7K, a high-quality benchmark derived from GRIT-20M. They used Grounding-DINO for re-validation, CLIP for semantic filtering, and performed extensive manual cleaning to ensure the layout-caption pairs are accurate and challenging.
Key Insights and Ideas
- The "Aha!" Moment: The researchers realized that the "object missing" problem occurs because the model's attention is spread too thin across a complex global latent space. By giving each object its own "branch" to grow in, the model is forced to account for every specified box.
- Weight Sharing is Enough: You don't need a unique set of weights for every object. Sharing weights across branches allows the model to use its general understanding of "how to draw an object" while the specific input (caption + box) provides the necessary local guidance.
- Mask-Based Fusion: Simple summation or averaging of features often leads to blurry results. Using explicit masks during the fusion step ensures that the high-fidelity features from the object branches land exactly where they are supposed to in the final image.
Key Results
HiCo demonstrates a clear lead over previous champions like GLIGEN and InstanceDiff:
- Spatial Accuracy: On the HiCo-7K benchmark, HiCo achieved a LocalIoU Score of 59.98, significantly higher than GLIGEN (48.54) and InstanceDiff (51.58), proving it follows bounding boxes much more faithfully.
- Image Quality: HiCo reached an FID of 14.24 (lower is better) on HiCo-7K, outperforming all other tested models in terms of visual fidelity and realism.
- Human Preference: In a multi-participant study, HiCo was preferred over other models across all dimensions, including object quantity, spatial location, and overall aesthetic quality (scoring 78.08 overall vs. GLIGEN's 67.76).
- Efficiency: Despite the multi-branch approach, HiCo's parallel inference mode makes it one of the fastest models in its class, with the shortest inference time among the six models compared.
Limitations and Future Work
- Occlusion Logic: While HiCo handles overlapping boxes better than most, it doesn't yet have an explicit "depth" understanding. It sometimes struggles to decide which object should be in front if the prompt doesn't explicitly state it (e.g., "A is behind B"). The authors are currently working on a depth-aware dataset to solve this.
- Complex Multi-LoRA: While compatible with LoRA, the model still faces challenges when trying to inject multiple distinct, complex LoRA-based concepts (like specific art styles or characters) into a single crowded layout simultaneously.
- Dataset Bias: Like all diffusion models, HiCo is limited by its training data; it performs best on natural scenes and may require further fine-tuning for highly abstract or technical diagrams.
Summary updated