Skeleton-Aware 3D Human Shape Reconstruction From Point Clouds

Haiyong Jiang, Jianfei Cai, Jianmin Zheng

Introduction

Reconstructing high-quality 3D human meshes from raw point clouds is a fundamental challenge in computer vision. While depth sensors like the Kinect have made point cloud acquisition straightforward, converting this noisy, irregular data into a clean, articulated human model remains difficult. This "inverse problem" is complicated by the high dimensionality of human meshes, the non-rigid nature of human movement, and the significant "parameter gap" between raw geometric points and the structured parameters of models like SMPL (Skinned Multi-Person Linear model).

Previous approaches often treated point clouds as generic shapes, ignoring the underlying biological structure—the skeleton—that governs human form and motion. This paper introduces a skeleton-aware framework that bridges the gap between unordered point data and structured parametric models using an intermediate skeleton representation and graph convolutional networks (GCNs).

The Skeleton-Aware Framework

The authors propose an end-to-end pipeline designed to map raw point clouds to SMPL parameters by explicitly modeling the human skeleton. This approach stabilizes the learning process by providing a strong structural prior.

The proposed end-to-end pipeline for skeleton-aware 3D human shape reconstruction, featuring Graph Aggregation (GA), an Attention Module (AM), and a Skeleton Graph Module (SGM).

Key Architectural Components

The pipeline consists of three novel modules:

  1. Modified PointNet++ with Graph Aggregation (GA): Unlike standard PointNet++ which processes points independently, the GA module builds a local k-nearest-neighbor (KNN) graph for each point. It employs spectral graph convolutions to capture local geometric patterns more effectively than simple pooling.
  2. Attention Module (AM): This module serves as the bridge between unordered point features and the ordered skeleton joints. The AM dynamically learns which point features contribute most to specific joints, mapping geometric data into an ordered set of 24 joint features.
  3. Skeleton Graph Module (SGM): Traditional methods often use a Multi-Layer Perceptron (MLP) to estimate SMPL parameters. The SGM instead treats the 24 joints as nodes in a graph, using graph convolutions to exploit natural kinematic dependencies (e.g., the relationship between the wrist, elbow, and shoulder).
Comparison between a standard MLP-based parameter estimation (a) and the proposed Skeleton Graph Module (b), which leverages graph convolutions to capture kinematic dependencies.

Training and Results

The model is initially trained in a supervised manner on synthetic datasets like SURREAL. To address the lack of ground truth in real-world scenarios, the authors implement an unsupervised online fine-tuning step. This step utilizes Chamfer distance to minimize the difference between the reconstructed mesh and the input point cloud, effectively bridging the "sim-to-real" gap.

Performance Highlights

  • State-of-the-Art Accuracy: On the Dyna dataset, the method achieved a mean error of 14.2mm for complex motions like jumping jacks, significantly outperforming previous benchmarks (e.g., 3DCODED at 17.2mm).
  • Ablation Insights: Removing the GA module increased error by over 4mm, confirming the importance of local point interactions.
  • Qualitative Quality: Visual results demonstrate smoother surfaces and more natural joint articulations compared to non-parametric or non-skeleton-aware methods.

Conclusion and Future Work

By integrating a skeleton prior into the reconstruction pipeline, this framework achieves superior stability and accuracy in 3D human shape estimation. While highly effective, the model currently faces limitations in capturing gender-specific nuances due to SMPL's parameter constraints and cannot yet reconstruct clothing or hair. Future research may extend the Graph Aggregation module to other point cloud tasks, such as semantic segmentation, to further leverage local geometric relationships.

Summary updated

Abstract

This work addresses the problem of 3D human shape reconstruction from point clouds. Considering that human shapes are of high dimensions and with large articulations, we adopt the state-of-the-art parametric human body model, SMPL, to reduce the dimension of learning space and generate smooth and valid reconstruction. However, SMPL parameters, especially pose parameters, are not easy to learn because of ambiguity and locality of the pose representation. Thus, we propose to incorporate skeleton awareness into the deep learning based regression of SMPL parameters for 3D human shape reconstruction. Our basic idea is to use the state-of-the-art technique PointNet++ to extract point features, and then map point features to skeleton joint features and finally to SMPL parameters for the reconstruction from point clouds. Particularly, we develop an end-to-end framework, where we propose a graph aggregation module to augment PointNet++ by extracting better point features, an attention module to better map unordered point features into ordered skeleton joint features, and a skeleton graph module to extract better joint features for SMPL parameter regression. The entire framework network is first trained in an end-to-end manner on synthesized dataset, and then online fine-tuned on unseen dataset with unsupervised loss to bridges gaps between training and testing. The experiments on multiple datasets show that our method is on par with the state-of-the-art solution.
BibTeX
@inproceedings{jiang2019skeleton,
  author = {Haiyong Jiang and Jianfei Cai and Jianmin Zheng},
  title = {{Skeleton-Aware 3D Human Shape Reconstruction From Point Clouds}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00553},
}
Figure