HCMA: Hierarchical Cross-model Alignment for Grounded Text-to-Image Generation

Hang Wang, Zhi-Qi Cheng, Chenhao Lin, Chao Shen, Lei ZhangarXiv 2025 · DOI · arXiv

Introduction

While modern text-to-image (T2I) models like Stable Diffusion excel at generating high-quality imagery, they often fail to respect explicit spatial constraints provided in natural language prompts. For instance, a request for "a cat on the left and a dog on the right" may result in blended features or incorrect positioning. "Grounded" generation attempts to solve this by incorporating bounding boxes, but existing methods frequently struggle to maintain a balance between global scene coherence and local object fidelity.

Methodology

The proposed Hierarchical Cross-Modal Alignment (HCMA) framework addresses these shortcomings by fundamentally altering the diffusion sampling process. Unlike previous methods that merely inject layout conditions into attention layers, HCMA introduces a dual-alignment mechanism that operates at every diffusion step.

The sampling loop is divided into two alternating phases: Alignment and Denoising. During the Alignment phase, a Vision Transformer (ViT) processes the noisy latent image to facilitate two distinct modules:

  1. Caption-to-Image Alignment (C2IA): This global-level module aligns a high-level [CLS] token from the latent with the CLIP embedding of the full text prompt, ensuring the overall scene remains semantically consistent.
  2. Object-to-Region Alignment (O2RA): This local-level module uses patch-level features fused with Fourier-encoded bounding box coordinates. These are aligned with CLIP embeddings of specific object category labels to enforce spatial constraints.
The HCMA sampling process interleaves global and local semantic consistency checks with standard denoising steps.

By calculating the gradients of these alignment losses, the model updates the latent representation before the standard U-Net denoising step. This "align-then-denoise" routine ensures that the generated image respects both the narrative of the prompt and the geometry of the layout.

Experimental Setup

HCMA was trained and evaluated using the MS-COCO 2014 dataset, a benchmark for grounded generation due to its extensive bounding-box annotations. The model was trained on approximately 82,000 images and evaluated on a 30,000-image validation subset. Performance was measured using Fréchet Inception Distance (FID) for realism and CLIP scores for text-image alignment.

Results and Discussion

HCMA achieves state-of-the-art results on the MS-COCO validation set, significantly outperforming baselines such as GLIGEN and BoxDiff.

  • Quantitative Performance: HCMA reached an FID of 8.74, a 3.33-point improvement over GLIGEN SD-v1.4. It also achieved a superior CLIP score of 0.3231.
  • Local Fidelity: Region-level metrics (FID: 12.48, CLIP: 0.2392) confirm that objects within bounding boxes are both realistic and semantically accurate.
  • Qualitative Superiority: HCMA effectively resolves common issues like the "counting problem" and "feature bleeding." For example, it can accurately generate a specific number of objects (e.g., five donuts) and maintain distinct identities for similar adjacent objects (e.g., an adult and child elephant).
Qualitative comparison showing HCMA's superior ability to follow complex grounding instructions compared to existing baselines.

Ablation studies highlighted the synergy between the two modules: while local alignment ensures spatial accuracy, global alignment acts as the "glue" that integrates these objects naturally into the scene.

Conclusion

HCMA introduces a robust hierarchical alignment strategy that significantly improves grounded text-to-image generation. By enforcing consistency at both global and local levels throughout the sampling process, it achieves unprecedented spatial control and visual realism. Future work will explore additional control modalities, such as depth maps, and adaptive weighting mechanisms to further refine the balance between global and local constraints.

Summary updated

Abstract

Text-to-image synthesis has progressed to the point where models can generate visually compelling images from natural language prompts. Yet, existing methods often fail to reconcile high-level semantic fidelity with explicit spatial control, particularly in scenes involving multiple objects, nuanced relations, or complex layouts. To bridge this gap, we propose a Hierarchical Cross-Modal Alignment (HCMA) framework for grounded text-to-image generation. HCMA integrates two alignment modules into each diffusion sampling step: a global module that continuously aligns latent representations with textual descriptions to ensure scene-level coherence, and a local module that employs bounding-box layouts to anchor objects at specified locations, enabling fine-grained spatial control. Extensive experiments on the MS-COCO 2014 validation set show that HCMA surpasses state-of-the-art baselines, achieving a 0.69 improvement in Frechet Inception Distance (FID) and a 0.0295 gain in CLIP Score. These results demonstrate HCMA's effectiveness in faithfully capturing intricate textual semantics while adhering to user-defined spatial constraints, offering a robust solution for semantically grounded image generation. Our code is available at https://github.com/hwang-cs-ime/HCMA.
BibTeX
@misc{wang2025hcma,
  author = {Hang Wang and Zhi-Qi Cheng and Chenhao Lin and Chao Shen and Lei Zhang},
  title = {{HCMA: Hierarchical Cross-model Alignment for Grounded Text-to-Image Generation}},
  year = {2025},
  eprint = {2505.06512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.06512},
}
Figure