Giving a Hand to Diffusion Models: A Two-Stage Approach to Improving Conditional Human Image Generation

Anton Pelykh, Ozge Mercanoglu Sincan, Richard Bowden

Abstract

Despite the rapid advancement of diffusion-based image synthesis, generating anatomically correct human hands remains a persistent challenge. This paper introduces a novel two-stage "divide and conquer" framework designed to mitigate common artifacts such as distorted fingers and incorrect joint counts. By decoupling hand generation from full-body synthesis and employing a specialized sequential mask expansion technique, the proposed method achieves a 92.3% improvement in hand pose accuracy over current state-of-the-art baselines.

Introduction

In the current landscape of generative AI, models like Stable Diffusion and Midjourney produce breathtaking landscapes and faces, yet frequently stumble when rendering human hands. These "nightmare fuel" artifacts—ranging from six-fingered hands to impossible joint articulations—stem from two primary factors: the high anatomic complexity of hands and a significant data imbalance in large-scale datasets. Hands possess numerous degrees of freedom and often self-occlude, making them statistically difficult to model. Furthermore, fine-tuning models on specialized hand datasets often leads to "catastrophic forgetting," where the model loses its ability to generate diverse styles or coherent full-body structures.

To address these issues, this research proposes a two-stage pipeline that treats hand generation as a specialized task, ensuring anatomical precision before integrating the hands into a broader human context.

Methodology

The proposed framework operates in two distinct stages: specialized hand generation followed by contextual body outpainting.

Stage 1: Specialized Hand Generation

Rather than attempting to generate a full person in a single pass, the system first focuses exclusively on the hands. The researchers trained a hand generator within a multi-task learning framework. This model does not merely predict the image pixels; it simultaneously predicts a semantic segmentation mask of the hand. To guide the generation, the model is conditioned on an 11-channel heatmap: 10 channels representing individual fingers (to better manage overlaps) and one channel for the global hand mask. This multi-tasking approach acts as a structural regularizer, forcing the model to adhere to realistic anatomical constraints.

Stage 2: Body Outpainting and Sequential Mask Expansion

Once high-fidelity hands are generated, they are aligned with a global body skeleton. A second model—an adapted ControlNet—is then used to "outpaint" the rest of the body and background around the pre-existing hands.

The primary technical challenge in this two-stage approach is the transition zone between the pre-generated hands and the newly generated arms. To prevent "boxy" artifacts or disconnected wrists, the authors developed Sequential Mask Expansion (SME). During the diffusion denoising process, the system begins with a large, dilated mask that covers the hand and the surrounding area. As the $T$ diffusion steps progress, the mask is gradually shrunk back to the original hand shape. This technique allows the model to "heal" the borders, ensuring that clothing, skin tones, and anatomy flow naturally from the arm into the pre-generated hand region.

Data Strategy

To train these models effectively, the authors curated a diverse training pipeline:

  • Hand Generation: Training utilized InterHand2.6M (studio-quality data), Re:InterHand (synthetic 3D renders), and HaGRID (diverse, real-world images).
  • Outpainting: The researchers used a filtered version of LAION-Human, which was re-captioned using LLaVA-v1.5-7b to provide the descriptive prompts necessary for high-quality contextual generation.

Results and Key Insights

The two-stage approach yielded significant quantitative improvements over established models like ControlNet and HumanSD:

  • Pose Accuracy: The model demonstrated a 92.3% improvement in hand Distance-based Average Precision (DAP) and a 30.5% improvement for full-body poses.
  • Error Reduction: The Mean Per Joint Position Error (MPJPE) for hands was reduced by 40% compared to the best-performing baselines.
  • Visual Quality: Using "Foreground FID" (a metric focusing on the subject rather than background noise), the method showed a 22% improvement in overall visual quality.

These results highlight several key insights. First, isolating complex components like hands allows the model to bypass the resolution bottlenecks of standard latent spaces (such as Stable Diffusion’s $64 \times 64$ grid). Second, varying the mask size during the denoising process (SME) is a highly effective way to leverage the inherent "healing" properties of diffusion models for seamless image composition.

Limitations and Future Work

While the framework represents a major step forward, certain limitations remain. The model is dependent on the quality of the input skeleton; if the skeleton lacks clear arm-to-wrist connections, the model may generate "floating" hands. Additionally, very small hands in wide-angle shots still face resolution limits imposed by the underlying VAE latent space. Future iterations may explore moving toward a shared latent space for both stages to reduce the computational overhead currently required by pixel-to-latent space conversions.

Summary updated

Abstract

Recent years have seen significant progress in human image generation, particularly with the advancements in diffusion models. However, existing diffusion methods encounter challenges when producing consistent hand anatomy and the generated images often lack precise control over the hand pose. To address this limitation, we introduce a novel approach to pose-conditioned human image generation, dividing the process into two stages: hand generation and subsequent body outpainting around the hands. We propose training the hand generator in a multi-task setting to produce both hand images and their corresponding segmentation masks, and employ the trained model in the first stage of generation. An adapted ControlNet model is then used in the second stage to outpaint the body around the generated hands, producing the final result. A novel blending technique is introduced to preserve the hand details during the second stage that combines the results of both stages in a coherent way. This involves sequential expansion of the outpainted region while fusing the latent representations, to ensure a seamless and cohesive synthesis of the final image. Experimental evaluations demonstrate the superiority of our proposed method over state-of-the-art techniques, in both pose accuracy and image quality, as validated on the HaGRID dataset. Our approach not only enhances the quality of the generated hands but also offers improved control over hand pose, advancing the capabilities of pose-conditioned human image generation. The source code is available here. 1 1 https://github.com/apelykh/hand-to-diffusion
BibTeX
@inproceedings{pelykh2024giving,
  author = {Anton Pelykh and Ozge Mercanoglu Sincan and Richard Bowden},
  title = {{Giving a Hand to Diffusion Models: A Two-Stage Approach to Improving Conditional Human Image Generation}},
  booktitle = {FG},
  year = {2024},
  url = {https://openreview.net/forum?id=EfKQZ5Bpmh},
}
Figure