Abstract
Despite the rapid advancement of diffusion-based image synthesis, generating anatomically correct human hands remains a persistent challenge. This paper introduces a novel two-stage "divide and conquer" framework designed to mitigate common artifacts such as distorted fingers and incorrect joint counts. By decoupling hand generation from full-body synthesis and employing a specialized sequential mask expansion technique, the proposed method achieves a 92.3% improvement in hand pose accuracy over current state-of-the-art baselines.
Introduction
In the current landscape of generative AI, models like Stable Diffusion and Midjourney produce breathtaking landscapes and faces, yet frequently stumble when rendering human hands. These "nightmare fuel" artifacts—ranging from six-fingered hands to impossible joint articulations—stem from two primary factors: the high anatomic complexity of hands and a significant data imbalance in large-scale datasets. Hands possess numerous degrees of freedom and often self-occlude, making them statistically difficult to model. Furthermore, fine-tuning models on specialized hand datasets often leads to "catastrophic forgetting," where the model loses its ability to generate diverse styles or coherent full-body structures.
To address these issues, this research proposes a two-stage pipeline that treats hand generation as a specialized task, ensuring anatomical precision before integrating the hands into a broader human context.
Methodology
The proposed framework operates in two distinct stages: specialized hand generation followed by contextual body outpainting.
Stage 1: Specialized Hand Generation
Rather than attempting to generate a full person in a single pass, the system first focuses exclusively on the hands. The researchers trained a hand generator within a multi-task learning framework. This model does not merely predict the image pixels; it simultaneously predicts a semantic segmentation mask of the hand. To guide the generation, the model is conditioned on an 11-channel heatmap: 10 channels representing individual fingers (to better manage overlaps) and one channel for the global hand mask. This multi-tasking approach acts as a structural regularizer, forcing the model to adhere to realistic anatomical constraints.
Stage 2: Body Outpainting and Sequential Mask Expansion
Once high-fidelity hands are generated, they are aligned with a global body skeleton. A second model—an adapted ControlNet—is then used to "outpaint" the rest of the body and background around the pre-existing hands.
The primary technical challenge in this two-stage approach is the transition zone between the pre-generated hands and the newly generated arms. To prevent "boxy" artifacts or disconnected wrists, the authors developed Sequential Mask Expansion (SME). During the diffusion denoising process, the system begins with a large, dilated mask that covers the hand and the surrounding area. As the $T$ diffusion steps progress, the mask is gradually shrunk back to the original hand shape. This technique allows the model to "heal" the borders, ensuring that clothing, skin tones, and anatomy flow naturally from the arm into the pre-generated hand region.
Data Strategy
To train these models effectively, the authors curated a diverse training pipeline:
- Hand Generation: Training utilized InterHand2.6M (studio-quality data), Re:InterHand (synthetic 3D renders), and HaGRID (diverse, real-world images).
- Outpainting: The researchers used a filtered version of LAION-Human, which was re-captioned using LLaVA-v1.5-7b to provide the descriptive prompts necessary for high-quality contextual generation.
Results and Key Insights
The two-stage approach yielded significant quantitative improvements over established models like ControlNet and HumanSD:
- Pose Accuracy: The model demonstrated a 92.3% improvement in hand Distance-based Average Precision (DAP) and a 30.5% improvement for full-body poses.
- Error Reduction: The Mean Per Joint Position Error (MPJPE) for hands was reduced by 40% compared to the best-performing baselines.
- Visual Quality: Using "Foreground FID" (a metric focusing on the subject rather than background noise), the method showed a 22% improvement in overall visual quality.
These results highlight several key insights. First, isolating complex components like hands allows the model to bypass the resolution bottlenecks of standard latent spaces (such as Stable Diffusion’s $64 \times 64$ grid). Second, varying the mask size during the denoising process (SME) is a highly effective way to leverage the inherent "healing" properties of diffusion models for seamless image composition.
Limitations and Future Work
While the framework represents a major step forward, certain limitations remain. The model is dependent on the quality of the input skeleton; if the skeleton lacks clear arm-to-wrist connections, the model may generate "floating" hands. Additionally, very small hands in wide-angle shots still face resolution limits imposed by the underlying VAE latent space. Future iterations may explore moving toward a shared latent space for both stages to reduce the computational overhead currently required by pixel-to-latent space conversions.
Summary updated