Abstract
Current state-of-the-art in 3D human pose and shape recovery relies on deep neural networks and statistical morphable body models, such as the Skinned Multi-Person Linear model (SMPL). However, regardless of the advantages of having both body pose and shape, SMPL-based solutions have shown difficulties to predict 3D bodies accurately. This is mainly due to the unconstrained nature of SMPL, which may generate unrealistic body meshes. Because of this, regression of SMPL parameters is a difficult task, often addressed with complex regularization terms. In this paper we propose to embed SMPL within a deep model to accurately estimate 3D pose and shape from a still RGB image. We use CNN-based 3D joint predictions as an intermediate representation to regress SMPL pose and shape parameters. Later, 3D joints are reconstructed again in the SMPL output. This module can be seen as an autoencoder where the encoder is a deep neural network and the decoder is SMPL model. We refer to this as SMPL reverse (SMPLR). By implementing SMPLR as an encoder-decoder we avoid the need of complex constraints on pose and shape. Furthermore, given that in-the-wild datasets usually lack accurate 3D annotations, it is desirable to lift 2D joints to 3D without pairing 3D annotations with RGB images. Therefore, we also propose a denoising autoencoder (DAE) module between CNN and SMPLR, able to lift 2D joints to 3D and partially recover from structured error. We evaluate our method on SURREAL and Human3.6M datasets, showing improvement over SMPL-based state-of-the-art alternatives by about 4 and 25 millimeters, respectively.
Introduction
Recovering the 3D shape and pose of a human from a single 2D image is a fundamental challenge in computer vision. While significant progress has been made, transitioning from 2D pixels to a fully realized 3D mesh remains unstable. The standard tool for this task is the SMPL (Skinned Multi-Person Linear) model, which describes the human body using shape parameters ($\beta$) and pose parameters ($\theta$).
Mapping an image directly to these parameters is an "ill-posed" problem because SMPL is a many-to-one function; different parameter combinations can result in similar-looking bodies. Furthermore, these parameters are highly sensitive to noise, where small errors can lead to anatomically impossible poses. Most existing methods rely on complex, hand-tuned regularization to maintain stability. In their paper, Madadi et al. propose reversing the model into an autoencoder framework to address these issues.
Methodology: The SMPLR Pipeline
The authors introduce a multi-stage pipeline that treats the SMPL model as a fixed decoder within a larger neural network.
- Volumetric CNN: The process begins with a Volumetric Stacked Hourglass Network (SHN) that predicts 3D heatmaps for joints and sparse surface landmarks. A soft-argmax layer converts these heatmaps into coordinates, maintaining differentiability.
- Denoising Autoencoder (DAE): Initial 3D estimates often contain "structured noise" (e.g., incorrect limb proportions due to occlusion). The DAE filters these estimates, learning plausible human proportions to "clean" the 3D points.
- SMPL Reverse (SMPLR): This core innovation uses a two-branch Multi-layer Perceptron (MLP) to take cleaned 3D points and regress the SMPL $\beta$ and $\theta$ parameters.
- The Decoder: These parameters are fed into the standard SMPL model to render the final mesh. Because the system is end-to-end differentiable, errors can be backpropagated from the final mesh to the initial CNN.
Key Insights
- Joints as a Geometric Anchor: By predicting 3D joints and landmarks first, the network uses a "bottleneck" that is easier to learn than abstract rotation matrices.
- Surface Landmarks: SMPLR utilizes 18 additional surface landmarks beyond standard skeletons. Without these, the model struggles to estimate body shape ($\beta$), often defaulting to an average body type.
- Implicit Regularization: Embedding SMPL as a decoder allows the model to learn its own constraints. The geometry of the SMPL model enforces anatomical plausibility during training without manual regularization.
- Lifting 2D to 3D: The DAE module effectively "lifts" 2D joint detections to 3D, allowing the model to function even when depth information is missing from the input.
Experimental Results
The model was evaluated on the SURREAL, Human3.6M, and UP-3D datasets. The results demonstrate a significant improvement in precision over previous SMPL-based methods.
- Human3.6M: SMPLR achieved a Mean Per Joint Position Error (MPJPE) improvement of 25mm over previous state-of-the-art SMPL-based methods like HMR.
- SURREAL: The model improved surface error by 4mm compared to multi-task learning baselines.
- Gender Accuracy: The system achieved 89.5% accuracy in gender estimation, which is critical as SMPL uses different templates for male and female bodies.
- Efficiency: The implementation runs at approximately 3 FPS on a single NVIDIA GTX 1080 Ti, making it viable for near-real-time applications.
Conclusion
SMPLR provides a robust framework for 3D human recovery by leveraging an autoencoder structure and intermediate 3D representations. While highly effective, the authors noted that gender misestimation remains a primary source of error. Future work may involve integrating temporal data from video to improve stability and utilizing denser landmark sets for finer detail in muscle and skin deformation.
Summary updated