MGHanD: Multi-modal Guidance for authentic Hand Diffusion

Taehyeon Eum, Jieun Choi, Tae-Kyun KimarXiv 2025 · OpenReview

TL;DR

This paper introduces MGHanD, a novel multi-modal guidance framework for diffusion models that significantly improves the realism and accuracy of generated human hands. By combining visual and textual guidance with a dynamic hand mask, MGHanD refines hand generation without altering the core diffusion model, leading to superior results in text-to-image synthesis.

What Problem Does This Paper Address?

Text-to-image (T2I) diffusion models have made incredible strides in generating realistic images from textual descriptions. However, a persistent Achilles' heel for these models is the generation of human hands. Hands are incredibly complex anatomical structures with numerous joints and degrees of freedom, making them notoriously difficult to render accurately. This often results in malformed hands with incorrect finger counts, distorted shapes, and unnatural poses, even when the rest of the image is photorealistic. This limitation not only impacts visual realism but also has implications for fields like robotics, where accurate hand representations are crucial for manipulation and interaction. Existing methods often struggle with consistency, especially when hands are deformed or in complex interactions, highlighting the need for a more robust solution.

Methods

MGHanD tackles the hand generation problem by introducing a multi-modal guidance system that operates during inference, meaning it doesn't require retraining the entire diffusion model. The core components are:

  • Visual Guidance via Discriminator: The authors train a discriminator on a custom dataset of real and generated hand images with captions. This discriminator learns to distinguish between realistic and unrealistic hand features. During inference, it provides a "visual nudge" to the diffusion process, pushing it towards more anatomically correct hand structures.
  • Textual Guidance via LoRA Adapter: Inspired by ConceptSlider, they use a LoRA (Low-Rank Adaptation) adapter. This adapter is trained to steer the diffusion model's generation from generic terms like "hands" to more specific, refined prompts like "natural hands" or "anatomically correct fingers." This allows for fine-grained control over hand attributes.
  • Cumulative Hand Mask: To ensure that the guidance is precisely applied to the hand region without affecting the rest of the image, MGHanD employs a dynamically evolving hand mask. This mask, generated by a hand detection model (Mediapipe), gradually expands over diffusion steps. This cumulative approach helps maintain the overall image coherence and style while focusing refinement on the hand.

These components are integrated into the diffusion model's denoising process, allowing for precise control over hand generation without compromising the model's original generative capabilities.

Datasets

The paper utilizes existing datasets for training and evaluation, and also describes the creation of a custom dataset for the visual guidance discriminator:

  • 100DOH [12]: A dataset featuring hands and hand-object interactions in first-person and third-person perspectives.
  • HAGRID [11]: A large third-person dataset containing images across 18 hand gesture classes.
  • Custom Dataset: Created by pairing real and generated hand images with captions. Captions were generated using the LLaVa [13] model, and Stable Diffusion was used to create "fake" data. This dataset is specifically for training the Hand Discriminator.

Key Insights and Ideas

  • Guidance at Inference is Key: The paper demonstrates that effective hand refinement can be achieved by applying guidance during the diffusion process, without the need for extensive fine-tuning of the base diffusion model. This preserves the model's original capabilities and reduces computational overhead.
  • Multi-Modal Synergy: The combination of visual (discriminator) and textual (LoRA) guidance is crucial. Visual guidance enforces anatomical correctness, while textual guidance allows for semantic control over hand attributes.
  • Dynamic Masking for Coherence: The cumulative hand mask is a clever way to isolate the refinement to the hand region. This prevents unintended side effects on the background or other objects, a common issue in image editing.
  • Addressing the "Hand Problem" Directly: Instead of relying on generic T2I models that struggle with hands, MGHanD specifically targets this weakness with specialized guidance mechanisms.

Key Results

  • Quantitative Improvements: MGHanD achieved the lowest FID and KID scores among the compared methods, indicating superior image quality and diversity. It also demonstrated the highest Hand Confidence and Hand Probability scores, signifying more accurate and likely hand detection in generated images. CLIP similarity was also competitive, showing good prompt alignment.
  • Qualitative Excellence: Visual comparisons show MGHanD producing significantly more realistic and anatomically correct hands compared to Stable Diffusion, ConceptSlider, and HandRefiner. It effectively handles complex poses and interactions, and corrects issues like malformed fingers and unnatural thumbs.
  • User Study Dominance: In a user study, MGHanD was overwhelmingly preferred for both visual quality and prompt alignment, outperforming other models by a significant margin.

Limitations and Future Work

The authors acknowledge a couple of limitations:

  • Inference Time: The current generation time of approximately 2 minutes per image is influenced by the guidance process in latent space. Optimizing inference efficiency is a key area for future work.
  • Minor Artifacts: In some cases, minor artifacts might appear, such as unintended object alterations when mask ranges expand, or occasional challenges in precisely controlling the number of hands.

Future work will focus on:

  • Refining masking techniques for more precise guidance control.
  • Optimizing model efficiency to reduce generation time.
  • Developing a more robust framework that aligns generated images more closely with user expectations.

Summary updated

Abstract

Diffusion-based methods have achieved significant successes in T2I generation, providing realistic images from text prompts. Despite their capabilities, these models face persistent challenges in generating realistic human hands, often producing images with incorrect finger counts and structurally deformed hands. MGHanD addresses this challenge by applying multi-modal guidance during the inference process. For visual guidance, we employ a discriminator trained on a dataset comprising paired real and generated images with captions, derived from various hand-in-the-wild datasets. We also employ textual guidance with LoRA adapter, which learns the direction from `hands' towards more detailed prompts such as `natural hands', and `anatomically correct fingers' at the latent level. A cumulative hand mask which is gradually enlarged in the assigned time step is applied to the added guidance, allowing the hand to be refined while maintaining the rich generative capabilities of the pre-trained model. In the experiments, our method achieves superior hand generation qualities, without any specific conditions or priors. We carry out both quantitative and qualitative evaluations, along with user studies, to showcase the benefits of our approach in producing high-quality hand images.
BibTeX
@misc{eum2025mghand,
  author = {Taehyeon Eum and Jieun Choi and Tae-Kyun Kim},
  title = {{MGHanD: Multi-modal Guidance for authentic Hand Diffusion}},
  year = {2025},
  eprint = {2503.08133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08133},
}
Figure