NIMBLE: A Non-rigid Hand Model with Bones and Muscles

Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Yuyao Zhang, Nianyi Li, Yuexin Ma, Lan Xu, Jingyi Yu

TL;DR

This paper introduces NIMBLE, a novel non-rigid parametric hand model that goes beyond traditional skeleton-only approaches by incorporating bones, muscles, and skin. By enforcing anatomical and kinematic rules, NIMBLE achieves unprecedented realism in animating 3D hands and can be used for generating realistic hand appearances and improving learning-based hand inference tasks.

What Problem Does This Paper Address?

The paper addresses the challenge of creating realistic and accurate 3D hand models, which is crucial for emerging applications like the Metaverse. Current state-of-the-art models primarily focus on the hand's skeletal structure, neglecting the complex interplay of muscles, tendons, and skin that dictates realistic movement and appearance. This omission leads to a lack of realism, especially in intricate gestures and deformations. The authors aim to bridge this gap by developing a comprehensive parametric model that accounts for the biomechanical intricacies of the human hand.

Methods

NIMBLE is built upon a volumetric template hand model that includes 20 bones (triangular meshes), 7 muscle groups (tetrahedral meshes), and a skin mesh. The core methodology involves:

  • Data Annotation: The authors extend the existing MRI-Hand dataset by annotating muscle and skin segmentation masks in addition to the provided bone annotations.
  • Multi-Stage Registration: A physically-based, multi-stage registration process is employed to align the template hand model with individual poses and subjects from the MRI data. This process includes pose initialization using simplified models and iterative refinement with various energy terms (geometry, regularization, elasticity, collision, attachment) to ensure accurate and physically plausible deformations.
  • Parametric Model Learning: After registration, a parametric model is learned through an iterative process of shape registration and parameter learning. This involves disentangling pose and shape deformations using a multi-stage strategy and incorporating penalty terms to prevent collisions and enforce physically correct muscle movements.
  • Appearance Modeling: A photometric capture system, "HandStage," is used to acquire high-quality textures, including diffuse, normal, and specular maps, to model realistic skin appearance, wrinkles, and palm prints. A parametric appearance model is then learned using PCA.
  • Pose Augmentation: To address the limited pose variance in MRI datasets, the model is further optimized using scans from the MANO dataset.

Datasets

  • MRI-Hand Dataset: Used for initial annotation of muscles and skins, and for training the core parametric hand model.
  • MANO Dataset: Utilized for pose augmentation to increase the model's ability to generalize to a wider range of hand poses.
  • FreiHAND Dataset: Used for evaluating hand inference tasks.

Key Insights and Ideas

  • Muscles are Crucial for Realism: The paper strongly emphasizes that realistic hand deformation and appearance cannot be achieved by modeling only bones and skin. The inclusion of muscle groups, and their interaction with bones and skin, is essential for capturing subtle bulges, wrinkles, and the overall naturalness of hand movements.
  • Anatomical and Kinematic Constraints Matter: By enforcing anatomical and kinematic rules during registration and parameter learning, NIMBLE achieves a higher level of realism than models that solely rely on surface fitting or simplified skinning.
  • Parametric Models as a Bridge: NIMBLE serves as a powerful tool not only for generating realistic hand animations but also for improving downstream tasks like hand pose and shape estimation. Its differentiable nature allows it to be seamlessly integrated into learning-based inference pipelines.
  • Photometric Capture for Appearance: The use of a dedicated photometric capture system (HandStage) is key to obtaining high-fidelity textures that capture fine details like wrinkles and palm prints, significantly enhancing the visual realism of rendered hands.

Key Results

  • Registration Accuracy: Quantitative results show that the registration process achieves a median per-vertex distance error of generally less than 2mm for MRI data and around 1mm for MANO skin data.
  • Model Compactness and Generalization: The learned shape and pose spaces demonstrate good compactness and generalization capabilities, with a reasonable number of components capturing a high percentage of variability.
  • Comparison with MANO: NIMBLE shows improved performance in capturing anatomically correct inner deformations and skin details compared to MANO, especially when fitting to MRI data. While MANO performs better on its own test set, NIMBLE with pose augmentation achieves comparable results on both MRI and MANO test sets.
  • Ablation Studies: Including muscle layers in the model leads to lower error metrics compared to bone-skin only models, demonstrating the positive impact of muscle modeling on both visual realism and fitting accuracy.
  • Qualitative Results: The paper presents numerous qualitative results showcasing realistic hand synthesis, animation, and rendering with diverse poses, shapes, and appearances, including the simulation of skin wrinkles due to muscle contraction.

Limitations and Future Work

  • Interconnections Not Explicitly Modeled: The authors acknowledge that while they model the interactions between bones, muscles, and skin, the explicit interconnections between these tissues are not fully modeled. Future work aims to address this by incorporating implicit skinning and explicit constraints.
  • Limited Hand Poses in MRI: The reliance on MRI data, which has limited pose variations due to acquisition costs, necessitates pose augmentation using other datasets like MANO.
  • Tendons and Ligaments: The current model does not explicitly include tendons and ligaments, which could further improve skin deformation and hand movement realism.
  • Secondary Deformations: Extending the model to include parametric secondary deformations (e.g., using blend shapes or FEM soft body dynamics) is a future direction.
  • Two-Handed Interaction: The current model primarily focuses on single-hand modeling. Future work aims to extend this to two-handed interactions and hand-object manipulation.
  • Synthetic Dataset Generation: The authors plan to leverage NIMBLE to create a high-quality, comprehensive synthetic hand dataset for training deep learning models for hand motion capture.

Summary updated

Abstract

Emerging Metaverse applications demand reliable, accurate, and photorealistic reproductions of human hands to perform sophisticated operations as if in the physical world. While real human hand represents one of the most intricate coordination between bones, muscle, tendon, and skin, state-of-the-art techniques unanimously focus on modeling only the skeleton of the hand. In this paper, we present NIMBLE, a novel parametric hand model that includes the missing key components, bringing 3D hand model to a new level of realism. We first annotate muscles, bones and skins on the recent Magnetic Resonance Imaging hand (MRI-Hand) dataset and then register a volumetric template hand onto individual poses and subjects within the dataset. NIMBLE consists of 20 bones as triangular meshes, 7 muscle groups as tetrahedral meshes, and a skin mesh. Via iterative shape registration and parameter learning, it further produces shape blend shapes, pose blend shapes, and a joint regressor. We demonstrate applying NIMBLE to modeling, rendering, and visual inference tasks. By enforcing the inner bones and muscles to match anatomic and kinematic rules, NIMBLE can animate 3D hands to new poses at unprecedented realism. To model the appearance of skin, we further construct a photometric HandStage to acquire high-quality textures and normal maps to model wrinkles and palm print. Finally, NIMBLE also benefits learning-based hand pose and shape estimation by either synthesizing rich data or acting directly as a differentiable layer in the inference network.
BibTeX
@article{li2022nimble,
  author = {Yuwei Li and Longwen Zhang and Zesong Qiu and Yingwenqi Jiang and Yuyao Zhang and Nianyi Li and Yuexin Ma and Lan Xu and Jingyi Yu},
  title = {{NIMBLE: A Non-rigid Hand Model with Bones and Muscles}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3528223.3530079},
}

Loading related content...

Figure