Pose2Pose: 3D Positional Pose-Guided 3D Rotational Pose Prediction for Expressive 3D Human Pose and Mesh Estimation

Gyeongsik Moon, Kyoung Mu LeearXiv 2020 · OpenReview

TL;DR

This paper introduces Hand4Whole, a novel system for whole-body 3D human mesh estimation that significantly improves the accuracy of 3D hand pose prediction. By introducing a Pose2Pose module that leverages hand MCP joint features for wrist rotations and strategically discarding body features for finger rotations, Hand4Whole achieves state-of-the-art results, particularly for complex hand articulations.

What Problem Does This Paper Address?

Estimating a complete 3D human mesh, encompassing the body, hands, and face simultaneously, is a challenging task. While progress has been made, accurately predicting the 3D pose of hands—with their intricate wrist and finger articulations—remains a significant hurdle. Existing methods often struggle with two key issues:

  1. Ignoring the Human Kinematic Chain for Wrist Prediction: The wrist's rotation is heavily influenced by the joints of the fingers' roots (MCP joints). Previous approaches haven't fully exploited this anatomical relationship, leading to less accurate wrist pose estimations.
  2. Over-reliance on Coarse Body Features for Finger Prediction: When predicting finger poses, prior methods often use general body features. However, these features are too coarse and lack the detailed information needed for precise finger articulation, especially since hands occupy small regions in images.

Methods

The core of Hand4Whole is its novel Pose2Pose module, designed to predict 3D joint rotations by first estimating 3D joint positions. This module offers two key innovations:

  • Positional Pose-Guided Pooling (PPP): Instead of global average pooling (GAP) on image features, PPP uses the predicted 3D joint positions to sample and interpolate features from a feature map. This allows the model to capture more localized and semantically rich information for each joint.
  • Leveraging MCP Joints for Wrist Rotation: For predicting 3D wrist rotations, Hand4Whole combines general body features with specific features from the MCP (Metacarpophalangeal) joints of the hands. The paper argues that MCP joints are crucial for determining wrist rotations due to their position in the kinematic chain. This is a departure from prior work that either ignored body features or used less informative hand features.

Hand4Whole also introduces a strategic approach to finger pose prediction:

  • Discarding Body Features for Finger Rotation: To avoid corrupting finger pose predictions with noisy and coarse body features, Hand4Whole's HandNet only uses fine-grained hand features for predicting 3D finger rotations. This allows for more accurate and detailed finger movements.

The overall system, Hand4Whole, integrates these components: a BodyNet (using Pose2Pose for body and wrist rotations), a HandNet (using Pose2Pose for finger rotations), and a FaceNet. These networks process cropped images of the respective body parts and feed their outputs into the SMPL-X layer for the final whole-body mesh.

Datasets

The paper utilizes several standard datasets for training and evaluation:

  • Training: Human3.6M, MSCOCO (whole-body version), MPII, and FreiHAND.
  • Evaluation:
    • Whole-body: EHF, AGORA.
  • Body-only: 3DPW.
  • Hand-only: FreiHAND.

Key Insights and Ideas

  • Anatomical Priors Matter: The paper strongly emphasizes the importance of incorporating anatomical knowledge, specifically the role of MCP joints in wrist rotation, to improve 3D pose estimation accuracy.
  • Feature Granularity is Key: The success of Hand4Whole hinges on using the right features for the right task. Detailed MCP features are vital for wrist rotation, while general body features are detrimental to finger rotation prediction.
  • Strategic Feature Discarding: Sometimes, the best approach is to remove potentially noisy or irrelevant information. By discarding body features for finger prediction, Hand4Whole avoids introducing errors.
  • Pose2Pose as a General Framework: The Pose2Pose module, with its PPP, proves to be a versatile component that can be applied to both body and hand pose estimation, demonstrating its effectiveness in extracting semantically rich joint features.

Key Results

Hand4Whole demonstrates significant improvements across various benchmarks:

  • Superior Hand Pose Accuracy: The paper reports substantial reductions in MPVPE (Mean Per-Vertex Position Error) for hands, particularly on the EHF dataset, where their "Body + MCP joints (Ours)" variant achieved an MPVPE of 39.8, outperforming other combinations.
  • State-of-the-Art Whole-Body Performance: Hand4Whole consistently outperforms previous whole-body methods on benchmarks like EHF, AGORA, and 3DPW, especially in metrics related to hand accuracy. For instance, on EHF, their PA MPVPE for "All Hands" was 76.8, significantly better than ExPose (219.8) and FrankMocap (218.0).
  • Qualitative Improvements: Visualizations (Figures 2, 5, 6, and A/B in the supplementary material) clearly show more plausible and accurate 3D hand poses, especially in challenging scenarios like occlusions or complex finger movements, compared to existing methods.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of ResNet and multiple networks suggests a non-trivial computational cost. The paper mentions downsampling for BodyNet to save computation, hinting at potential trade-offs.
  • Dataset Specificity: While trained on diverse datasets, the performance on unseen or significantly different data distributions might be a future area of exploration.
  • Future Work: The paper's success with MCP joints suggests that further exploration of specific anatomical relationships within the human body could lead to even more accurate estimations. The Pose2Pose framework itself could be extended to other related tasks.

Summary updated

Abstract

Whole-body 3D human mesh estimation aims to reconstruct the 3D human body, hands, and face simultaneously. Although several methods have been proposed, accurate prediction of 3D hands, which consist of 3D wrist and fingers, still remains challenging due to two reasons. First, the human kinematic chain has not been carefully considered when predicting the 3D wrists. Second, previous works utilize body features for the 3D fingers, where the body feature barely contains finger information. To resolve the limitations, we present Hand4Whole, which has two strong points over previous works. First, we design Pose2Pose, a module that utilizes joint features for 3D joint rotations. Using Pose2Pose, Hand4Whole utilizes hand MCP joint features to predict 3D wrists as MCP joints largely contribute to 3D wrist rotations in the human kinematic chain. Second, Hand4Whole discards the body feature when predicting 3D finger rotations. Our Hand4Whole is trained in an end-to-end manner and produces much better 3D hand results than previous whole-body 3D human mesh estimation methods. The codes are available here at https://github.com/mks0601/Hand4Whole_RELEASE.
BibTeX
@misc{moon2020pose,
  author = {Gyeongsik Moon and Kyoung Mu Lee},
  title = {{Pose2Pose: 3D Positional Pose-Guided 3D Rotational Pose Prediction for Expressive 3D Human Pose and Mesh Estimation}},
  year = {2020},
  eprint = {2011.11534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.11534},
}
Figure