Learnable Human Mesh Triangulation for 3D Human Pose and Shape Estimation

Sungho Chun, Sungbum Park, Ju Yong ChangWACV 2023

TL;DR

This paper introduces a novel two-stage approach, Learnable Human Mesh Triangulation (LMT), for reconstructing 3D human meshes from multi-view images. LMT improves upon existing methods by first estimating mesh vertices directly and then fitting the SMPL model, leading to more accurate joint rotation and shape estimation, while also addressing computational challenges and multi-view inconsistencies.

What Problem Does This Paper Address?

Estimating the 3D pose and shape of humans from images is a long-standing problem in computer vision. While significant progress has been made in estimating 3D joint positions, accurately determining joint rotations and the overall human shape has been more challenging. Existing methods often fall into two categories: those that estimate joints first and then fit a human model (like SMPL) to these joints, or those that directly regress SMPL parameters from images using deep learning. Both approaches have limitations. The former struggles with the ambiguity of joint rotations and shape due to insufficient information from just joint positions, often requiring heavy regularization. The latter faces difficulties in learning the highly non-linear mapping from images to SMPL parameters. This paper aims to bridge this gap by proposing a method that leverages mesh vertex information for more robust estimation of rotations and shape, while also being more amenable to learning.

Methods

The proposed LMT method employs a two-stage pipeline:

  1. Vertex Regression: Instead of directly regressing SMPL parameters or estimating 3D joints, LMT first estimates the 3D coordinates of mesh vertices. This is achieved through a CNN backbone that processes multi-view image features. A key innovation here is the use of a "visibility module" that estimates per-vertex visibility from each input image. This visibility information is then used to augment the image features before they are aggregated in a 3D volumetric space. This aggregation process, followed by a vertex regression module using 3D convolutions and soft-argmax, outputs the 3D coordinates of a subsampled mesh. The use of subsampled vertices is a crucial optimization to manage the high memory requirements of 3D heatmaps for dense meshes.
  2. SMPL Fitting: Once the mesh vertices are estimated, the SMPL model is fitted to these vertices. This fitting module optimizes SMPL parameters (pose, shape, and translation) to minimize the difference between the fitted SMPL mesh and the regressed vertices. This stage leverages the richer information provided by surface vertices compared to just joint positions, leading to improved estimation of joint rotations and human shape.

The visibility module is implemented using the I2L-MeshNet method for single-view mesh reconstruction, and its output is further subsampled to reduce computational load. The feature aggregation module uses a cuboid of a fixed size centered on the pelvis to ensure the target human is contained. The vertex regression module uses an encoder-decoder structure with 3D convolutions. The fitting module uses an optimization process with data and regularization terms, including an exponential regularization term to prevent unnatural joint bending.

Datasets

The paper uses the following datasets for training and evaluation:

  • Human3.6M [10]: A large-scale dataset for 3D human pose estimation, used for training and testing. Ground-truth SMPL meshes are derived from this dataset.
  • MPI-INF-3DHP [26]: A dataset for 3D human pose estimation. For this work, only multi-view training data was used, as the test data includes single-view images. Ground-truth SMPL parameters are not provided, so pseudo ground-truth meshes were used for training.

Key Insights and Ideas

  • Surface Vertices over Joints for Fitting: Fitting the SMPL model to estimated mesh vertices, rather than just 3D joint positions, provides richer information for resolving ambiguities in joint rotation and shape estimation.
  • Sub-vertex Estimation for Efficiency: Directly predicting all 6890 SMPL vertices is computationally prohibitive due to large 3D heatmaps. Estimating a smaller set of "sub-vertices" significantly reduces memory usage while maintaining or even improving performance.
  • Per-Vertex Visibility for Robustness: Incorporating per-vertex visibility information from individual views helps to mitigate multi-view inconsistencies caused by occlusions, leading to more accurate vertex estimation.
  • Heatmap Regression for Learnability: Extending the heatmap-based keypoint estimation framework to mesh vertex estimation makes the learning process more tractable compared to direct SMPL parameter regression, which suffers from high non-linearity.

Key Results

  • Improved Rotation and Shape: LMT significantly outperforms previous methods in terms of joint rotation and human shape estimation on the Human3.6M dataset.
  • Competitive Joint Location: LMT achieves competitive performance in 3D joint location estimation.
  • Efficiency Gains: Sub-vertex estimation drastically reduces GPU memory requirements, allowing for training with larger heatmap resolutions. For example, using 108 sub-vertices instead of 6890 full vertices reduces memory by approximately 64 times.
  • Resolution Matters: A heatmap resolution of 64x64x64 yielded the best performance on Human3.6M, achieving better results than full-vertex models without increased memory cost.
  • Visibility Benefits: Experiments show that using per-vertex visibility improves MPVE (Mean-per-vertex-error) and leads to better MPJPE (Mean-per-joint-position-error) and angular distance results, even after fitting.
  • Cross-Dataset Generalization: LMT demonstrates strong generalization capabilities, outperforming a softmax baseline on the MPI-INF-3DHP dataset after being trained on Human3.6M.

Limitations and Future Work

  • Visibility Module Complexity: The authors acknowledge that using a single-view mesh reconstruction model (I2L-MeshNet) to obtain visibility complicates the proposed model. Future work could focus on developing more efficient methods for acquiring visibility information.
  • In-the-Wild Scenarios: While the method shows good generalization, further investigation into diverse viewpoints and truly "in-the-wild" input images is suggested for future work.

Summary updated

BibTeX
@inproceedings{chun2023learnable,
  author = {Sungho Chun and Sungbum Park and Ju Yong Chang},
  title = {{Learnable Human Mesh Triangulation for 3D Human Pose and Shape Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure