Abstract
Estimating accurate 3D human meshes from single 2D images remains a significant challenge, particularly in scenarios involving complex poses and heavy occlusion. We introduce SEFD (SMPL Edge Feature Distillation), a novel framework that leverages knowledge distillation to improve structural boundary estimation. By training a student model to mimic a teacher equipped with ground-truth structural information, SEFD enables robust 3D reconstruction from noisy real-world inputs without increasing inference-time complexity. Our method achieves state-of-the-art performance on benchmarks such as 3DPW and OCHuman.
Introduction
3D human mesh estimation is a cornerstone of digital human modeling and action recognition. While current models perform well on standard upright poses, they frequently fail when joints reach extreme ranges of motion (complex poses) or when body parts are hidden by objects or other people (occlusion). Existing methods often rely on 2D pose detectors that struggle when structural information is missing or ambiguous. SEFD addresses these hurdles by teaching the model to "imagine" clean structural boundaries from noisy real-world images.
Methodology: SMPL Edge Feature Distillation
The core of SEFD is a "Teacher-Student" distillation strategy designed to bridge the gap between perfect structural knowledge and noisy real-world data.
1. SMPL Edge Map Generation
The authors developed a tool to generate ground-truth edge maps from 3D SMPL (Skinned Multi-Person Linear) model parameters. Unlike simple outlines, these are overlapping edge maps that explicitly encode depth ordering—indicating which body parts are in front of others. To ensure consistency across scales, an Adaptive Dilation technique scales edge thickness based on the subject's size in the image.
2. Feature Distillation
- The Teacher: Trained using the original image paired with the "perfect" SMPL overlapping edge map.
- The Student: Receives the original image plus a "noisy" edge map generated by a standard Canny edge detector.
- The Objective: Through a specialized $Log-\sigma-L_1$ loss function, the student's encoder is forced to produce feature representations that mimic the teacher’s. This allows the student to extract high-quality structural features even when input edges are messy or incomplete.
Datasets and Evaluation
The model was trained on standard datasets including Human3.6M, MuCo-3DHP, MSCOCO, and MPII. Evaluation focused on challenging, occlusion-heavy benchmarks:
- 3DPW: Including variants 3DPW-OCC and 3DPW-PC.
- OCHuman: Focused on heavy occlusion.
- CrowdPose and LSP: For multi-person and diverse pose scenarios.
To support future research, the authors provided new occlusion and complex pose annotations for these datasets, defining "complex poses" formally based on joints exceeding their typical Range of Motion (ROM).
Key Results
SEFD demonstrates clear quantitative and qualitative superiority over existing state-of-the-art (SOTA) methods.
- Quantitative Gains: On the 3DPW dataset, SEFD improved Mean Per-Joint Position Error (MPJPE) by 2.8% and Mean Per-Vertex Position Error (MPVPE) by 1.9%.
- Occlusion Robustness: On the OCHuman dataset, SEFD significantly outperformed specialized models like OCHMR and 3DCrowdNet, proving that distilled edge features effectively "see through" occlusions.
- Qualitative Fidelity: Visual results confirm that SEFD correctly places limbs in extreme poses (e.g., deep squats or climbing) where baseline models often produce tangled or impossible meshes.
Conclusion
SEFD provides a robust framework for 3D human mesh estimation by using edges as a structural prior. By distilling knowledge from perfect SMPL edge maps into a student model, the system gains the ability to handle complex poses and occlusions using only raw images and simple edge detectors at test time. Future work will focus on refining edge detection in extremely low-contrast or blurry environments to further bridge the domain gap between lab data and "in-the-wild" imagery.
Summary updated