Robust Photo-Realistic Hand Gesture Generation: from Single View to Multiple View

Qifan Fu, Xu Chen, Muhammad Asad, Shanxin Yuan, Changjae Oh, Gregory Slabaugh

TL;DR

Generating realistic hands in AI images is notoriously difficult due to complex finger movements and self-occlusion. This paper introduces MUFEN, a multi-modal framework that leverages multi-view 3D mesh priors—rendering hands from optimal complementary angles—to give diffusion models a complete spatial understanding of the hand. By fusing multi-view geometry with text, depth, and bounding box localization, MUFEN achieves state-of-the-art photo-realistic hand generation with significantly fewer structural errors.

What Problem Does This Paper Address?

Hands have an extraordinary number of degrees of freedom, making them one of the biggest hurdles in human-centric image generation. Current state-of-the-art diffusion models frequently fail, producing hands with missing, mutated, or extra fingers.

While existing approaches try to fix this by providing 3D hand meshes or depth maps as structural priors, they almost exclusively rely on a single viewpoint (usually the front view). When fingers are occluded behind the palm or other fingers, a single 2D projection loses vital 3D topological information. This ambiguity leaves the diffusion model guessing, resulting in anatomical distortions.

Figure 1: Photo-realistic hand gesture generation via various methods. Our method, named MUFEN, improves the realism and accuracy of gesture generation by fusing multi-view and multi-modal features of the gesture.

Methods

The authors propose MUFEN (Multi-Modal UNet-based Feature Encoder), which tackles occlusion by looking at the hand from multiple angles simultaneously.

  1. Multi-View Mesh Rendering: Using an off-the-shelf 3D hand estimator (HaMeR), the framework extracts the 3D MANO mesh of the hand. It then applies coordinate transformations to render the hand from six orthogonal views: front, rear, left, right, top, and bottom. Instead of feeding all six views (which introduces redundancy), it dynamically selects the most informative complementary pair based on the largest projected area.
  2. Dual-Stream Rendering Encoder: This module processes the selected pair of mesh images through parallel ResNet50 backbones equipped with Convolutional Block Attention Modules (CBAM) to capture spatial and channel-wise features before fusing them.
  3. Multi-Modal Feature Fusion: MUFEN integrates multiple modalities: the multi-view meshes, text prompts (via CLIP), hand depth maps, and hand bounding boxes.
  4. Gated Bounding Box (BBox) Fusion Module: To ensure the model knows exactly where the hand is without complex region-cropping hacks, a novel BBox fusion module uses attention and learnable gating mechanisms to inject spatial localization data directly into each modality's feature representation.
Figure 2: Multi-view hand MANO mesh rendering pipeline.

Datasets

The model is trained and evaluated on the HaGRID v1 dataset, a popular benchmark for gesture recognition adapted for generation tasks. It contains 28,814 training images and 7,623 testing images across 18 distinct gesture categories (e.g., "Like", "Ok", "Peace"). The authors augmented this dataset by generating the 3D multi-view mesh annotations for all images using HaMeR.

Key Insights and Ideas

  • More views aren't always better: Intuitively, one might think feeding all six views would yield the best results. However, ablation studies revealed that using 4 or 6 views actually degraded performance due to information redundancy and overlap. Selecting just the single best co-axial view pair (2 views) struck the perfect balance.
  • Depth maps are crucial structural anchors: Removing the depth map caused the most severe performance drop in their ablation studies, proving that global depth context is vital even when high-fidelity local meshes are present.
  • Direct feature fusion via gating: Instead of using zero-padding strategies to align hand features with the full image space, the gated BBox fusion module allows multi-modal hand features to be directly injected into the diffusion model's latent space, preserving local context seamlessly.
Figure 6: The bounding box (BBox) fusion module adds lo- calization information from the gesture bounding box to the multimodal features extracted by MUFEN through attention and gating mechanisms.

Key Results

MUFEN achieves state-of-the-art results on the HaGRID v1 dataset, outperforming strong baselines like ControlNet, HandRefiner, and RealisHuman.

  • Quantitative Performance: It achieves a FID-Hand score of 26.85 (a significant drop from ControlNet's 32.21 and HanDrawer's 28.75) and a KID-Hand score of 0.0173, indicating much higher fidelity and anatomical accuracy in the generated hand regions.
  • Qualitative Improvements: Visual results show that MUFEN successfully resolves complex gestures (like "Ok" or "Three 2") where fingers frequently overlap, producing clear, distinct finger boundaries without classic "AI hand" mutations.
  • Efficiency: Inference takes 2.39 seconds per image on an NVIDIA A100 GPU, which is slightly faster than HanDrawer (2.45s) while delivering superior quality.

Limitations and Future Work

While the paper demonstrates robust improvements, some limitations remain:

  • Dependency on Upstream 3D Annotators: MUFEN relies heavily on the accuracy of the initial 3D mesh estimation tool (HaMeR). If the upstream annotator fails to capture the correct hand pose from a highly distorted or blurry image, MUFEN's multi-view priors will inherit those errors.
  • Computational Overhead of Pre-rendering: Generating six-view meshes takes about 1.45 seconds per image on a laptop GPU compared to 0.48 seconds for front-view-only rendering. Although this is a pre-processing step, optimizing the multi-view rendering pipeline remains an open area for future improvement.

Summary updated

Abstract

High-fidelity hand gesture generation represents a significant challenge in human-centric generation tasks. Existing methods typically employ a single-view mesh-rendered image prior to enhancing gesture generation quality. However, the spatial complexity of hand gestures and the inherent limitations of single-view rendering make it difficult to capture complete gesture information, particularly when fingers are occluded. The fundamental contradiction lies in the loss of 3D topological relationships through 2D projection and the incomplete spatial coverage inherent to single-view representations. Diverging from single-view prior approaches, we propose a multi-view prior framework, named Multi-Modal UNet-based Feature Encoder (MUFEN), to guide diffusion models in learning comprehensive 3D hand information. Specifically, we extend conventional front-view rendering to include rear, left, right, top, and bottom perspectives, selecting the most information-rich view combination as training priors to address occlusion. This multi-view prior with a dedicated dual stream encoder significantly improves the model's understanding of complete hand features. Furthermore, we design a bounding box feature fusion module, which can fuse the gesture localization features and multi-modal features to enhance the location-awareness of the MUFEN features to the gesture-related features. Experiments demonstrate that our method achieves state-of-the-art performance in both quantitative metrics and qualitative evaluations. The source code is available at https://github.com/fuqifan/MUFEN.
BibTeX
@inproceedings{fu2025robust,
  author = {Qifan Fu and Xu Chen and Muhammad Asad and Shanxin Yuan and Changjae Oh and Gregory Slabaugh},
  title = {{Robust Photo-Realistic Hand Gesture Generation: from Single View to Multiple View}},
  booktitle = {ACM Multimedia},
  year = {2025},
}
Figure