FuRPE: Learning Full-body Reconstruction from Part Experts

Zhaoxin Fan, Yuqing Pan, Hao Xu, Zhenbo Song, Zhicheng Wang, Kejian Wu, Hongyan Liu, Jun He

TL;DR

FuRPE (Full-body Reconstruction from Part Experts) addresses the severe scarcity of 3D full-body annotated data by leveraging specialized "part expert" models to generate high-quality pseudo-labels. By combining a rigorous pseudo-label selection scheme, feature distillation, and an Exponential Moving Average (EMA) student-teacher training strategy, FuRPE significantly improves 3D full-body mesh recovery from single images, setting a new state-of-the-art.

What Problem Does This Paper Address?

3D human body reconstruction has transitioned from estimating simple torso-and-limb meshes (like the SMPL model) to full-body estimation that includes expressive faces and detailed hands (the SMPL-X model). However, training these full-body models requires massive amounts of annotated data.

Capturing simultaneous ground truth for the face, hands, and body requires expensive optical or inertial motion capture tools, specialized gloves, and facial capture rigs. Harmonizing these disparate data streams into a single cohesive representation adds massive financial and logistical complexity. As a result, existing full-body datasets are small, severely limiting the generalization capabilities of deep learning models.

Methods

Instead of relying on expensive full-body annotations, FuRPE introduces a "divide-and-conquer" pseudo-labeling framework that bootstraps knowledge from existing specialized models.

  1. Part Experts: The framework utilizes three pre-trained, highly accurate models specialized in narrow domains: SPIN for the body, DECA for the face, and FrankMocap for the hands. These experts process cropped regions of unannotated images to generate initial pseudo-labels and features.
  2. Pseudo Ground-Truth Selection Scheme: To prevent low-quality or erroneous expert predictions from corrupting the training process, the authors implement a strict three-step filtering pipeline:
  3. Step 1: OpenPose detects 2D keypoints; images with fewer than 12 high-confidence keypoints are discarded.
  4. Step 2: Images resulting in invalid part-expert outputs are removed.
  5. Step 3: The predicted 3D parameters are projected back into 2D space and compared against the original OpenPose detections. If the Mean Squared Error (MSE) exceeds 1.5 cm, the image is discarded.
  6. EMA Self-Supervision & Feature Distillation: To further mitigate the inherent biases and noise in pseudo-labels, FuRPE uses an Exponential Moving Average (EMA) student-teacher framework. The teacher network guides the student network under different random data augmentations (rotation, translation, color enhancement, pixel noise). Additionally, an expert-derived feature distillation strategy uses Kullback-Leibler (KL) divergence to align the network's internal representations with the rich features generated by the part experts.

The complete training pipeline is illustrated below:

Figure 2: Pipeline of our work. (a) The training pipeline of using part-experts to generate supervision signals. (b) The training pipeline of Exponential Moving Average self-supervision.

Datasets

  • Training Data: A mix of indoor and outdoor datasets including CuratedFits, H3.6M, MPI, Ochuman, Posetrack, and EFT.
  • Evaluation Data:
  • EHF: Used for comprehensive full-body evaluation (Vertex-to-Vertex distance).
  • 3DPW: Used for body-only evaluation.
  • NoW: Used for face-only evaluation.
  • FreiHAND: Used for hand-only evaluation.

Key Insights and Ideas

  • Leveraging Asymmetric Expertise: Training a single network to master the body, face, and hands simultaneously from scratch is incredibly difficult. Bootstrapping the network using specialized models that excel in narrow domains provides a powerful, scalable alternative to manual annotation.
  • Strict Filtering Trumps Data Volume: The 3-step selection scheme is crucial. By aggressively discarding images where the projected 3D mesh doesn't align with 2D keypoints, the authors ensure the network learns from high-fidelity targets.
  • Unlocking the Power of Scaling: The authors demonstrate a clear "scaling law" behavior: as more unannotated datasets are added and pseudo-labeled through FuRPE, full-body reconstruction performance consistently improves. This proves the framework can successfully exploit massive, cheap, unannotated image datasets.

Key Results

  • Full-Body Superiority: On the full-body EHF dataset, FuRPE combined with the ExPose baseline achieved a Vertex-to-Vertex (V2V) Procrustes error of 50.6 mm, significantly outperforming the baseline ExPose (55.1 mm) and the state-of-the-art PIXIE model (55.0 mm).
  • Single-Stage Efficiency: A fully convolutional, single-stage variant (FuRPE+ResNet) achieved a V2V error of 54.8 mm. This is notable because it outperforms PIXIE without needing to explicitly crop out the face, hands, and body during inference.
  • Part-Specific Performance: On the 3DPW body reconstruction dataset, FuRPE achieved a Procrustes MPJPE of 59.0 mm, outperforming even body-centric models like SPIN (59.2 mm), while maintaining highly competitive performance on specialized face (NoW) and hand (FreiHAND) benchmarks.

Limitations and Future Work

  • Hand Resolution Bottleneck: While FuRPE shows massive improvements in body and face reconstruction, hand reconstruction performance remains constrained. The authors attribute this to the low resolution of hands within full-body training images, which limits the network's capacity to learn intricate finger details.
  • Future Work: The authors suggest that future iterations should focus on incorporating high-resolution images into the training and testing sets to specifically enhance hand reconstruction performance and benchmarking.

Summary updated

Abstract

In the field of full-body reconstruction, the scarcity of annotated data often impedes the efficacy of prevailing methods. To address this issue, we introduce FuRPE, a novel framework that employs part-experts and an ingenious pseudo ground-truth selection scheme to derive high-quality pseudo labels. These labels, central to our approach, equip our network with the capability to efficiently learn from the available data. Integral to FuRPE is a unique exponential moving average training strategy and expert-derived feature distillation strategy. These novel elements of FuRPE not only serve to further refine the model but also to reduce potential biases that may arise from inaccuracies in pseudo labels, thereby optimizing the network's training process and enhancing the robustness of the model. We apply FuRPE to train both two-stage and fully convolutional single-stage full-body reconstruction networks. Our exhaustive experiments on numerous benchmark datasets illustrate a substantial performance boost over existing methods, underscoring FuRPE's potential to reshape the state-of-the-art in full-body reconstruction.
BibTeX
@misc{fan2022furpe,
  author = {Zhaoxin Fan and Yuqing Pan and Hao Xu and Zhenbo Song and Zhicheng Wang and Kejian Wu and Hongyan Liu and Jun He},
  title = {{FuRPE: Learning Full-body Reconstruction from Part Experts}},
  year = {2022},
  eprint = {2212.00731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00731},
}
Figure