Abstract
Vanilla text-to-image diffusion models often struggle with anatomical consistency, leading to artifacts such as distorted limbs or incorrect poses. While auxiliary models like ControlNet mitigate these issues, they require additional user-provided inputs at inference time. This paper introduces the Human-centric Prior (HcP) layer, a modular "plug-and-play" component that embeds structural knowledge directly into the diffusion model's cross-attention mechanism. By utilizing a human-centric alignment loss during fine-tuning, the model learns to prioritize anatomical integrity from text prompts alone. Experimental results on the Human-Art and Laion-Human datasets demonstrate that HcP significantly improves image quality and anatomical accuracy compared to standard fine-tuning methods.
Introduction
Text-to-image (T2I) models like Stable Diffusion have revolutionized content creation but remain prone to "hallucinating" the human form. Common failures include impossible joint angles, spaghetti limbs, or incorrect numbers of digits. Existing solutions typically rely on external guidance (e.g., pose maps), which increases the complexity for the end-user. The proposed HcP layer addresses this by bridging the gap between ease of use (text-only prompts) and structural integrity, allowing the model to "understand" human structure internally.
Methodology
The core of the proposed approach is the integration of human-centric structural priors into the U-Net architecture during the training phase.
The Human-centric Prior (HcP) Layer
The HcP layer is a learnable module inserted into the cross-attention layers of the U-Net. It employs a Human-centric Token Extractor to identify relevant keywords in the prompt (e.g., "yoga," "jumping") and enhances their interaction with spatial latent features. This ensures that the model's attention is focused on the correct anatomical regions when generating human figures.
Human-centric Alignment Loss
To ground the model in physical reality, the authors utilize a Human-centric Alignment Loss. During training, a pre-trained ResNet-50 extracts structural features from reference pose images. The HcP layer is trained to align the model's internal attention maps with these features. This process effectively teaches the model the spatial "blueprint" associated with specific human actions.
Scale & Step Aware Learning
The researchers identified that human structure is primarily established during the early timesteps of the diffusion process and within mid-resolution layers. To exploit this, they implemented a Scale & Step Aware Learning strategy using dynamic loss weighting. This focuses training on the most impactful parts of the generation process, ensuring structural learning does not interfere with fine-grained texture generation in later stages.
Experimental Results
The HcP method was evaluated using the Human-Art (50k images) and Laion-Human (1M images) datasets, which provide diverse scenarios ranging from natural photography to classical paintings.
Quantitative Performance
HcP achieved a Fréchet Inception Distance (FID) of 28.71 on the Human-Art dataset, outperforming both vanilla Stable Diffusion (33.31) and standard LoRA fine-tuning (29.22). It also maintained high CLIP-Scores, indicating strong alignment between the generated images and text prompts.
Qualitative Analysis
Human evaluations focused on Anatomy Quality (AQ). HcP consistently produced more realistic figures in complex scenarios such as dance and acrobatics, where baseline models frequently failed to maintain joint connectivity or limb proportions.
Conclusion
The Human-centric Prior (HcP) layer provides an effective, inference-efficient way to improve human image generation. By baking structural awareness into the model's cross-attention maps, it eliminates the need for auxiliary pose inputs while maintaining high artistic quality. While challenges remain in complex multi-person scenes, HcP represents a significant step toward more reliable and user-friendly human image synthesis.
Summary updated