Abstract
3D human body reconstruction has advanced significantly, yet capturing the full body—including the face and hands—remains difficult due to a scarcity of comprehensive 3D datasets. FuRPE (Full-body Reconstruction from Part Experts) overcomes this data bottleneck by leveraging specialized "part-expert" models to generate high-quality pseudo-labels from unannotated images. By integrating a robust filtering mechanism and an Exponential Moving Average (EMA) teacher-student training framework, FuRPE establishes a new state-of-the-art in full-body mesh reconstruction from single images.
Introduction
While models like SMPL have enabled accurate torso and limb reconstruction, extending this to the face and hands is computationally and logistically demanding. High-fidelity full-body datasets require expensive motion-capture setups and complex synchronization, leading to a "data desert" for integrated models.
Existing methods typically rely on fully supervised training with limited ground-truth data. In contrast, FuRPE utilizes the wealth of unlabelled images by employing specialized experts for different body parts to provide supervision, effectively bypassing the need for manual 3D annotations.
Methodology
The FuRPE framework is built on the principle of learning from a panel of experts through a student-teacher paradigm. The core idea leverages the modularity of the SMPL-X model, which is mathematically composed of three sub-models: FLAME for the face, SMPL for the body, and MANO for the hands.
Part-Expert Supervision
The system utilizes three state-of-the-art (SOTA) models as experts: SPIN for the body, DECA for the face, and FrankMocap for the hands. These experts process unlabelled images to generate pseudo-labels. To ensure the quality of these labels, FuRPE implements a rigorous three-step filtering process:
- Confidence Check: Images with fewer than 12 high-confidence OpenPose keypoints are discarded.
- Validity Check: Logical consistency of expert outputs is verified.
- Re-projection Error: 3D parameters are projected back to 2D; if the Mean Squared Error (MSE) exceeds 1.5cm, the data is rejected.
Feature Distillation and EMA
Beyond raw coordinate supervision, FuRPE distills internal features from the expert models, capturing nuanced "knowledge" of part geometry. To stabilize training and mitigate noise from pseudo-labels, an Exponential Moving Average (EMA) strategy is used. A "Teacher" model (a moving average of the student's weights) provides self-supervision to the "Student" model, ensuring smoother convergence and better generalization.
Datasets and Evaluation
FuRPE was trained on a diverse collection of datasets, including CuratedFits, H3.6M, MPI, Ochuman, Posetrack, and EFT. Evaluation was conducted across several benchmarks:
- EFH: The primary metric for full-body reconstruction.
- 3DPW: Focused on body-only performance.
- NoW & FreiHAND: Specialized benchmarks for face and hand accuracy, respectively.
Key Results and Insights
FuRPE achieves a new state-of-the-art on the EFH dataset, reducing the Vertex-to-Vertex (V2V) error from 55.1mm to 50.6mm. On the 3DPW body benchmark, it achieved a Procrustes MPJPE of 59.0mm, outperforming both full-body (PIXIE) and body-centric (SPIN) models.
Key insights from the study include:
- Filtering is Critical: The quality of pseudo-labels, ensured by re-projection checks, is more impactful than sheer volume.
- One-Stage Efficiency: The "one-stage" ResNet-based model outperformed two-stage models that use crops, suggesting that global context is vital for local part reconstruction.
- Qualitative Superiority: FuRPE demonstrates improved handling of "closed-mouth" expressions and complex wrist articulations compared to previous baselines.
Conclusion and Future Work
FuRPE demonstrates that specialized part-experts can effectively bridge the data gap for full-body reconstruction. While hand reconstruction remains challenging due to low pixel resolution in full-body views, future iterations may incorporate high-resolution part-specific crops or temporal consistency for video data to further refine these details.
Summary updated