Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation

Junyan Wang, Zhenhong Sun, Zhiyu Tan, Xuanbai Chen, Weihua Chen, Hao Li, Cheng Zhang, Yang Song

Abstract

Vanilla text-to-image diffusion models often struggle with anatomical consistency, leading to artifacts such as distorted limbs or incorrect poses. While auxiliary models like ControlNet mitigate these issues, they require additional user-provided inputs at inference time. This paper introduces the Human-centric Prior (HcP) layer, a modular "plug-and-play" component that embeds structural knowledge directly into the diffusion model's cross-attention mechanism. By utilizing a human-centric alignment loss during fine-tuning, the model learns to prioritize anatomical integrity from text prompts alone. Experimental results on the Human-Art and Laion-Human datasets demonstrate that HcP significantly improves image quality and anatomical accuracy compared to standard fine-tuning methods.

Introduction

Text-to-image (T2I) models like Stable Diffusion have revolutionized content creation but remain prone to "hallucinating" the human form. Common failures include impossible joint angles, spaghetti limbs, or incorrect numbers of digits. Existing solutions typically rely on external guidance (e.g., pose maps), which increases the complexity for the end-user. The proposed HcP layer addresses this by bridging the gap between ease of use (text-only prompts) and structural integrity, allowing the model to "understand" human structure internally.

Methodology

The core of the proposed approach is the integration of human-centric structural priors into the U-Net architecture during the training phase.

The Human-centric Prior (HcP) Layer

The HcP layer is a learnable module inserted into the cross-attention layers of the U-Net. It employs a Human-centric Token Extractor to identify relevant keywords in the prompt (e.g., "yoga," "jumping") and enhances their interaction with spatial latent features. This ensures that the model's attention is focused on the correct anatomical regions when generating human figures.

Overview of the HcP architecture, including the token extractor and the alignment loss mechanism.

Human-centric Alignment Loss

To ground the model in physical reality, the authors utilize a Human-centric Alignment Loss. During training, a pre-trained ResNet-50 extracts structural features from reference pose images. The HcP layer is trained to align the model's internal attention maps with these features. This process effectively teaches the model the spatial "blueprint" associated with specific human actions.

Scale & Step Aware Learning

The researchers identified that human structure is primarily established during the early timesteps of the diffusion process and within mid-resolution layers. To exploit this, they implemented a Scale & Step Aware Learning strategy using dynamic loss weighting. This focuses training on the most impactful parts of the generation process, ensuring structural learning does not interfere with fine-grained texture generation in later stages.

Experimental Results

The HcP method was evaluated using the Human-Art (50k images) and Laion-Human (1M images) datasets, which provide diverse scenarios ranging from natural photography to classical paintings.

Quantitative Performance

HcP achieved a Fréchet Inception Distance (FID) of 28.71 on the Human-Art dataset, outperforming both vanilla Stable Diffusion (33.31) and standard LoRA fine-tuning (29.22). It also maintained high CLIP-Scores, indicating strong alignment between the generated images and text prompts.

Qualitative Analysis

Human evaluations focused on Anatomy Quality (AQ). HcP consistently produced more realistic figures in complex scenarios such as dance and acrobatics, where baseline models frequently failed to maintain joint connectivity or limb proportions.

Qualitative comparison between Stable Diffusion V1-5, LoRA, and the proposed HcP method across various dance poses.

Conclusion

The Human-centric Prior (HcP) layer provides an effective, inference-efficient way to improve human image generation. By baking structural awareness into the model's cross-attention maps, it eliminates the need for auxiliary pose inputs while maintaining high artistic quality. While challenges remain in complex multi-person scenes, HcP represents a significant step toward more reliable and user-friendly human image synthesis.

Summary updated

Abstract

Vanilla text-to-image diffusion models struggle with generating accurate human images, commonly resulting in imperfect anatomies such as unnatural postures or disproportionate limbs. Existing methods address this issue mostly by fine-tuning the model with extra images or adding additional control - human-centric priors such as pose or depth maps - during the image generation phase. This paper explores the integration of these human-centric priors directly into the model fine-tuning stage, essentially eliminating the need for extra conditions at the inference stage. We realize this idea by proposing a human-centric alignment loss to strengthen human-related information from the textual prompts within the cross-attention maps. To ensure semantic detail richness and human structural accuracy during fine-tuning, we introduce scale-aware and step-wise constraints within the diffusion process, according to an indepth analysis of the cross-attention layer. Extensive experiments show that our method largely improves over state-of-the-art text-to-image models to synthesize high-quality human images based on user-written prompts. Project page: https://hcplayercvpr2024.github.io.
BibTeX
@misc{wang2024towards,
  author = {Junyan Wang and Zhenhong Sun and Zhiyu Tan and Xuanbai Chen and Weihua Chen and Hao Li and Cheng Zhang and Yang Song},
  title = {{Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation}},
  year = {2024},
  eprint = {2403.05239},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.05239},
}
Figure