FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

Yu Rong, Takaaki Shiratori, Hanbyul Joo

TL;DR

FrankMocap tackles the challenge of estimating a full 3D human pose (body, face, and hands) from a single image. It achieves this by breaking down the problem into estimating each part independently and then intelligently integrating these estimates, offering a fast and accurate solution that outperforms existing methods.

What Problem Does This Paper Address?

Estimating a complete 3D human pose from a single 2D image is a complex task. While significant progress has been made in estimating individual components like the body, face, or hands, accurately capturing all of them simultaneously from a single image remains difficult. This is partly due to the scale differences (hands and faces are much smaller than the body) and the susceptibility of hands to motion blur and self-occlusion in real-world videos. Existing methods often struggle to integrate these parts seamlessly or require computationally intensive optimization, making them unsuitable for real-time applications. The lack of comprehensive datasets that capture all these body parts in diverse in-the-wild scenarios further exacerbates this challenge.

Methods

FrankMocap employs a modular approach:

  • Independent Part Regression: The system first uses specialized, state-of-the-art regression models to estimate 3D poses for the face, hands (left and right), and body independently. This allows each module to leverage existing expertise and datasets without compromise.
  • Integration Module: The core novelty lies in how these independent estimates are combined. FrankMocap proposes three integration strategies:
  • Copy-Paste: A fast, simple method that directly transfers joint angle parameters, with specific handling for wrist poses.
  • Optimization-Based: A more accurate but slower approach that fits the whole-body model (SMPL-X) to the individual part predictions and 2D keypoints.
  • Wrist Integration Network: A learned approach using a small neural network to adjust arm poses for better hand localization, balancing speed and precision. This network is trained on synthetically generated data.

The system leverages the SMPL-X parametric model for unified whole-body representation.

Datasets

The paper mentions using several datasets for training and evaluation:

  • Hand Module Training: FreiHAND, HO-3D, MTC, STB, and RHD. MPII+NZSL is used for validation only.
  • Body Module Training: Human3.6M and outdoor pseudo-GT datasets (EFT).
  • Face Module: Utilizes an off-the-shelf model (RingNet) trained on FLAME.
  • Whole-Body Evaluation: EHF dataset.
  • Body-Only Evaluation: 3DPW dataset.

Key Insights and Ideas

  • Modularity is Key: Breaking down the complex whole-body estimation into independent part estimation and then integrating them is a powerful strategy. This circumvents the need for massive, unified datasets and allows leveraging existing specialized models.
  • Integration is Crucial: The way individual part estimates are combined significantly impacts the final result. The paper explores a spectrum of trade-offs between speed and accuracy through different integration methods.
  • Motion Blur Robustness: Explicitly addressing motion blur through data augmentation is vital for real-world performance, especially for hand estimation.
  • Leveraging Existing Models: The approach allows for easy upgrades by simply swapping out individual part estimation modules with newer, better-performing ones.

Key Results

  • Hand Module Performance: FrankMocap's hand module is competitive with state-of-the-art hand-only methods on benchmarks like STB, RHD, and MPII+NZSL, achieving comparable or better AUC scores.
  • Body Module Performance: The body module, a fine-tuned version of existing models, performs comparably to state-of-the-art body-only methods on the 3DPW dataset.
  • Whole-Body Performance: FrankMocap significantly outperforms previous SMPL-X based whole-body methods (SMPLify-X, ExPose) on the EHF dataset across various metrics (V2V, PA-V2V). The optimization-based integration module generally achieves the best results.
  • Speed: The copy-paste and wrist integration network modules enable fast, near real-time performance, suitable for live demos.

Limitations and Future Work

  • Failure Cases: The paper acknowledges that severe occlusion and blur can still lead to failure cases, as shown in Figure 14.
  • Dataset Limitations: The body module's performance might be slightly degraded due to using fewer datasets compared to some prior works.
  • Future Work: The modular design inherently allows for future improvements by simply replacing individual modules with more advanced ones. The authors also suggest that their approach can be extended to leverage future algorithms for each part.

Summary updated

Abstract

3D human pose estimation can facilitate various applications, such as assistive technologies and AR/VR. Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body. In this paper, we present FrankMocap, a fast and accurate whole-body 3D pose estimation system that can produce 3D face, hands, and body simultaneously from in-the-wild monocular images. The idea of FrankMocap is its modular design: We first run 3D pose regression methods for face, hands, and body independently, followed by composing the regression outputs via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. We develop three different integration modules that trade off between latency and accuracy. All of them are capable of providing simple yet effective solutions to unify the separate outputs into seamless whole-body pose estimation results. We quantitatively and qualitatively demonstrate that our modularized system outperforms both the optimization-based and end-to-end methods of estimating whole-body pose.
BibTeX
@misc{rong2021frankmocap,
  author = {Yu Rong and Takaaki Shiratori and Hanbyul Joo},
  title = {{FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration}},
  year = {2021},
  eprint = {2108.06428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.06428},
}
Figure