Introduction
While modern text-to-image (T2I) models like Stable Diffusion excel at generating high-quality imagery, they often fail to respect explicit spatial constraints provided in natural language prompts. For instance, a request for "a cat on the left and a dog on the right" may result in blended features or incorrect positioning. "Grounded" generation attempts to solve this by incorporating bounding boxes, but existing methods frequently struggle to maintain a balance between global scene coherence and local object fidelity.
Methodology
The proposed Hierarchical Cross-Modal Alignment (HCMA) framework addresses these shortcomings by fundamentally altering the diffusion sampling process. Unlike previous methods that merely inject layout conditions into attention layers, HCMA introduces a dual-alignment mechanism that operates at every diffusion step.
The sampling loop is divided into two alternating phases: Alignment and Denoising. During the Alignment phase, a Vision Transformer (ViT) processes the noisy latent image to facilitate two distinct modules:
- Caption-to-Image Alignment (C2IA): This global-level module aligns a high-level
[CLS]token from the latent with the CLIP embedding of the full text prompt, ensuring the overall scene remains semantically consistent. - Object-to-Region Alignment (O2RA): This local-level module uses patch-level features fused with Fourier-encoded bounding box coordinates. These are aligned with CLIP embeddings of specific object category labels to enforce spatial constraints.
By calculating the gradients of these alignment losses, the model updates the latent representation before the standard U-Net denoising step. This "align-then-denoise" routine ensures that the generated image respects both the narrative of the prompt and the geometry of the layout.
Experimental Setup
HCMA was trained and evaluated using the MS-COCO 2014 dataset, a benchmark for grounded generation due to its extensive bounding-box annotations. The model was trained on approximately 82,000 images and evaluated on a 30,000-image validation subset. Performance was measured using Fréchet Inception Distance (FID) for realism and CLIP scores for text-image alignment.
Results and Discussion
HCMA achieves state-of-the-art results on the MS-COCO validation set, significantly outperforming baselines such as GLIGEN and BoxDiff.
- Quantitative Performance: HCMA reached an FID of 8.74, a 3.33-point improvement over GLIGEN SD-v1.4. It also achieved a superior CLIP score of 0.3231.
- Local Fidelity: Region-level metrics (FID: 12.48, CLIP: 0.2392) confirm that objects within bounding boxes are both realistic and semantically accurate.
- Qualitative Superiority: HCMA effectively resolves common issues like the "counting problem" and "feature bleeding." For example, it can accurately generate a specific number of objects (e.g., five donuts) and maintain distinct identities for similar adjacent objects (e.g., an adult and child elephant).
Ablation studies highlighted the synergy between the two modules: while local alignment ensures spatial accuracy, global alignment acts as the "glue" that integrates these objects naturally into the scene.
Conclusion
HCMA introduces a robust hierarchical alignment strategy that significantly improves grounded text-to-image generation. By enforcing consistency at both global and local levels throughout the sampling process, it achieves unprecedented spatial control and visual realism. Future work will explore additional control modalities, such as depth maps, and adaptive weighting mechanisms to further refine the balance between global and local constraints.
Summary updated