Handy: Towards a High Fidelity 3D Hand Shape and Appearance Model

Rolandos Alexandros Potamias, Stylianos Ploumpis, Stylianos Moschoglou, Vasileios Triantafyllou, Stefanos ZafeiriouCVPR 2023 · DOI

Abstract

Handy is a large-scale 3D hand model derived from over 1,200 diverse subjects, significantly expanding the capabilities of the industry-standard MANO model. By integrating a high-resolution shape mesh with a GAN-based appearance model, Handy reconstructs intricate details such as wrinkles, veins, and nail polish from single "in-the-wild" images.

Introduction

For years, the MANO model has served as the "gold standard" for 3D hand modeling. However, it suffers from three primary limitations: a lack of demographic diversity (trained on only 31 adults), low mesh resolution (778 vertices), and a lack of realistic texture. Existing texture models often rely on Principal Component Analysis (PCA), which produces "blurry" results lacking high-frequency details like pores or wrinkles. Handy addresses these gaps, providing a high-fidelity model capable of representing a vast range of human diversity and visual detail.

Methodology

The Handy model was developed through a multi-stage pipeline involving high-resolution 3D scans of 1,208 subjects.

  • Shape Modeling: The researchers created a high-resolution template with 8,407 vertices, using PCA to establish a linear basis for shape variations.
  • Appearance Modeling: To avoid the blurring inherent in PCA, a Style-based GAN was trained to generate high-resolution UV texture maps, preserving sharp, high-frequency details.
  • Reconstruction: A ResNet-50 encoder predicts pose, shape, and texture latent codes from images, optimized via a differentiable renderer and perceptual loss (LPIPS).
The high-resolution mesh template used in Handy features 8,407 vertices, a significant increase over the 778 vertices in the MANO model.

Datasets and Diversity

The researchers introduced the Handy Dataset, featuring 1,208 subjects ranging from infants to 81-year-olds. This diversity is crucial; the paper demonstrates that children are not merely "small adults," as their bone structures and proportions differ fundamentally. By including over 360 children, Handy is the first model to handle this demographic accurately. Additionally, a synthetic dataset of 30,000 images was used to bridge the domain gap between simulation and real-world photography.

Results and Realism

Handy achieves state-of-the-art performance across several benchmarks:

  • Demographic Accuracy: On children under 12, Handy reduced reconstruction error to 0.21mm, a nearly 4x improvement over MANO (0.77mm).
  • Texture Quality: In LPIPS benchmarks, Handy-GAN (0.010) significantly outperformed the previous best model, HTML (0.092).
  • Benchmark Performance: On the FreiHand benchmark, Handy achieved a Mean Per-Vertex Error (MPVPE) of 7.8mm, surpassing previous methods.
  • Qualitative Fidelity: The model successfully reconstructs specific features like tattoos, rings, and skin conditions such as vitiligo, which are beyond the reach of traditional parametric models.
Handy's GAN-based appearance model can reconstruct complex skin conditions like vitiligo, demonstrating its superior fidelity over previous models.

Discussion and Future Work

While Handy offers unprecedented fidelity, its high-resolution mesh and GAN-based generation are computationally intensive, potentially limiting real-time mobile applications. Extreme occlusion also remains a challenge for monocular reconstruction. Future work involves integrating Handy into full-body models like SMPL-X to create more detailed and realistic human avatars.

Summary updated

BibTeX
@inproceedings{potamias2023handy,
  author = {Rolandos Alexandros Potamias and Stylianos Ploumpis and Stylianos Moschoglou and Vasileios Triantafyllou and Stefanos Zafeiriou},
  title = {{Handy: Towards a High Fidelity 3D Hand Shape and Appearance Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00453},
}
Figure