FuRPE: Learning Full-body Reconstruction from Part Experts

Zhaoxin Fan, Yuqing Pan, Hao Xu, Zhenbo Song, Zhicheng Wang, Kejian Wu, Hongyan Liu, Jun He

Abstract

3D human body reconstruction has advanced significantly, yet capturing the full body—including the face and hands—remains difficult due to a scarcity of comprehensive 3D datasets. FuRPE (Full-body Reconstruction from Part Experts) overcomes this data bottleneck by leveraging specialized "part-expert" models to generate high-quality pseudo-labels from unannotated images. By integrating a robust filtering mechanism and an Exponential Moving Average (EMA) teacher-student training framework, FuRPE establishes a new state-of-the-art in full-body mesh reconstruction from single images.

Introduction

While models like SMPL have enabled accurate torso and limb reconstruction, extending this to the face and hands is computationally and logistically demanding. High-fidelity full-body datasets require expensive motion-capture setups and complex synchronization, leading to a "data desert" for integrated models.

Existing methods typically rely on fully supervised training with limited ground-truth data. In contrast, FuRPE utilizes the wealth of unlabelled images by employing specialized experts for different body parts to provide supervision, effectively bypassing the need for manual 3D annotations.

Comparison of training pipelines: (a) existing methods rely on scarce ground-truth, while (b) FuRPE uses part-experts to generate pseudo-labels.

Methodology

The FuRPE framework is built on the principle of learning from a panel of experts through a student-teacher paradigm. The core idea leverages the modularity of the SMPL-X model, which is mathematically composed of three sub-models: FLAME for the face, SMPL for the body, and MANO for the hands.

Part-Expert Supervision

The system utilizes three state-of-the-art (SOTA) models as experts: SPIN for the body, DECA for the face, and FrankMocap for the hands. These experts process unlabelled images to generate pseudo-labels. To ensure the quality of these labels, FuRPE implements a rigorous three-step filtering process:

  1. Confidence Check: Images with fewer than 12 high-confidence OpenPose keypoints are discarded.
  2. Validity Check: Logical consistency of expert outputs is verified.
  3. Re-projection Error: 3D parameters are projected back to 2D; if the Mean Squared Error (MSE) exceeds 1.5cm, the data is rejected.

Feature Distillation and EMA

Beyond raw coordinate supervision, FuRPE distills internal features from the expert models, capturing nuanced "knowledge" of part geometry. To stabilize training and mitigate noise from pseudo-labels, an Exponential Moving Average (EMA) strategy is used. A "Teacher" model (a moving average of the student's weights) provides self-supervision to the "Student" model, ensuring smoother convergence and better generalization.

The FuRPE architecture: (a) Part-Experts Supervision for pseudo-label generation and (b) EMA Self-supervision for robust training.

Datasets and Evaluation

FuRPE was trained on a diverse collection of datasets, including CuratedFits, H3.6M, MPI, Ochuman, Posetrack, and EFT. Evaluation was conducted across several benchmarks:

  • EFH: The primary metric for full-body reconstruction.
  • 3DPW: Focused on body-only performance.
  • NoW & FreiHAND: Specialized benchmarks for face and hand accuracy, respectively.

Key Results and Insights

FuRPE achieves a new state-of-the-art on the EFH dataset, reducing the Vertex-to-Vertex (V2V) error from 55.1mm to 50.6mm. On the 3DPW body benchmark, it achieved a Procrustes MPJPE of 59.0mm, outperforming both full-body (PIXIE) and body-centric (SPIN) models.

Key insights from the study include:

  • Filtering is Critical: The quality of pseudo-labels, ensured by re-projection checks, is more impactful than sheer volume.
  • One-Stage Efficiency: The "one-stage" ResNet-based model outperformed two-stage models that use crops, suggesting that global context is vital for local part reconstruction.
  • Qualitative Superiority: FuRPE demonstrates improved handling of "closed-mouth" expressions and complex wrist articulations compared to previous baselines.

Conclusion and Future Work

FuRPE demonstrates that specialized part-experts can effectively bridge the data gap for full-body reconstruction. While hand reconstruction remains challenging due to low pixel resolution in full-body views, future iterations may incorporate high-resolution part-specific crops or temporal consistency for video data to further refine these details.

Summary updated

Abstract

Full-body reconstruction is a fundamental but challenging task. Owing to the lack of annotated data, the performances of existing methods are largely limited. In this paper, we propose a novel method named Full-body Reconstruction from Part Experts~(FuRPE) to tackle this issue. In FuRPE, the network is trained using pseudo labels and features generated from part-experts. An simple yet effective pseudo ground-truth selection scheme is proposed to extract high-quality pseudo labels. In this way, a large-scale of existing human body reconstruction datasets can be leveraged and contribute to the model training. In addition, an exponential moving average training strategy is introduced to train the network in a self-supervised manner, further boosting the performance of the model. Extensive experiments on several widely used datasets demonstrate the effectiveness of our method over the baseline. Our method achieves the state-of-the-art performance. Code will be publicly available for further research.
BibTeX
@misc{fan2022furpe,
  author = {Zhaoxin Fan and Yuqing Pan and Hao Xu and Zhenbo Song and Zhicheng Wang and Kejian Wu and Hongyan Liu and Jun He},
  title = {{FuRPE: Learning Full-body Reconstruction from Part Experts}},
  year = {2022},
  eprint = {2212.00731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00731},
}
Figure