Abstract
Recovering 3D human pose and shape from a single RGB image is a challenging task, typically requiring large-scale datasets with precise 3D ground-truth labels. This paper presents a novel self-supervised framework that eliminates the need for such labels by leveraging automatically extracted 2D joints and depth maps as "proxy" supervision. A key innovation is an uncertainty-aware learning module that utilizes human body symmetry to estimate the reliability of these noisy proxy inputs. Our method achieves state-of-the-art performance among self-supervised approaches on standard benchmarks like Human3.6M and 3DPW, demonstrating the effectiveness of depth-based self-consistency and symmetry-driven uncertainty modeling.
Introduction
Reconstructing a 3D human mesh—capturing both the pose and the shape of a person—from a single 2D image is a cornerstone of computer vision with broad applications in AR/VR, human-robot interaction, and motion analysis. While fully supervised models have made significant strides, they are constrained by the difficulty and expense of capturing high-quality 3D annotations for "in-the-wild" images.
Existing self-supervised attempts often rely on additional constraints such as multi-view data, synthetic datasets, or temporal consistency in video sequences. This work addresses the more difficult problem of learning 3D structure from a single, unlabeled image. By treating the outputs of pre-trained 2D detectors and depth estimators as proxy labels, the proposed framework learns to reconstruct 3D meshes through internal data consistency. We argue that depth maps provide a richer spatial signal than traditional 2D silhouettes, and that the inherent symmetry of the human body can serve as a powerful supervisor for estimating the reliability of noisy inputs.
Methodology
The proposed pipeline consists of several integrated components designed to handle the noise inherent in self-supervised learning:
Proxy Input Generation and Trimming
The system first employs off-the-shelf, pre-trained models to extract 2D joints and a depth map from the input image. These serve as the initial "pseudo-labels." To improve the quality of the depth signal, we introduce Depth Map Trimming (TIQR). This method uses an Interquartile Range (IQR) approach to filter out extreme outliers and noise from the raw depth estimates, providing a cleaner target for the model.
Uncertainty-Aware Module
Because proxy inputs are often ambiguous due to lighting or occlusions, the model cannot trust them equally. The core of our framework is an Uncertainty-Aware Module that learns a reliability weight for each body joint. This module operates on the principle of symmetry consistency: it compares the predicted 3D structure against the proxy inputs. If the estimated depth differences between symmetric body parts (e.g., left vs. right limbs) are anatomically inconsistent, the module flags those areas as uncertain and assigns them a lower weight in the loss function.
Self-Consistency Loss
The model predicts parameters for the SMPL (Skinned Multi-Person Linear) model to generate a 3D mesh. This mesh is then projected back into the 2D image plane. The final loss function minimizes the discrepancy between these projections and the proxy inputs, with the contribution of each joint weighted by its learned uncertainty score. This ensures the model prioritizes reliable information while remaining robust to noisy proxy data.
Experiments and Results
Datasets
The model was evaluated on three major benchmarks: Human3.6M (indoor with precise 3D ground truth), 3DPW (challenging outdoor scenes), and UP-3D (outdoor variety). Crucially, 3D labels were used only for evaluation, not for training.
Key Performance Metrics
The proposed method significantly outperformed previous self-supervised state-of-the-art (SOTA) methods:
- 3DPW: Achieved a PA-MPJPE of 89.5 mm, a substantial improvement over the previous best of 102.7 mm.
- Human3.6M: Reached a PA-MPJPE of 85.4 mm, compared to the prior SOTA of 90.5 mm.
Ablation studies highlighted the importance of the individual components. Removing the Uncertainty-Aware module and Depth Trimming caused the reconstruction error to increase from 159.4 mm to 208.2 mm MPJPE in specific tests, confirming their critical role in stabilizing self-supervised learning.
Conclusion and Future Work
This research demonstrates that high-quality 3D human mesh recovery is possible without manual 3D annotations by leveraging depth maps and anatomical symmetry. The use of depth as a "richer substitute" for silhouettes and the introduction of symmetry-based uncertainty weighting are key drivers of the model's success.
Despite these gains, limitations remain. The model struggles with severe occlusions where proxy inputs become entirely unreliable. Additionally, the current reliance on a sparse set of joints leads to less detailed reconstruction of hands and feet. Future work will explore incorporating more fine-grained keypoints and temporal information to further enhance the robustness and detail of the recovered meshes.
Summary updated