3D Human Pose and Shape Estimation Through Collaborative Learning and Multi-View Model-Fitting

Zhongguo Li, Magnus Oskarsson, Anders HeydenWACV 2021

TL;DR

This paper introduces a novel self-supervised framework for 3D human pose and shape estimation that leverages multi-view images. By collaboratively training a deep neural network with a multi-view optimization approach, the method achieves more accurate and robust estimations than previous single-view or purely regression-based techniques.

What Problem Does This Paper Address?

Estimating the 3D pose and shape of a human body from images is a fundamental yet challenging problem in computer vision, with applications ranging from augmented reality to animation. Traditional approaches fall into two main categories: optimization-based methods that fit a parametric human body model (like SMPL) to image cues, and regression-based methods that use deep neural networks to directly predict pose and shape from images.

Optimization methods can be accurate but are often slow and sensitive to initialization. Regression methods are fast but require large amounts of annotated data and can struggle with ambiguity, especially from single-view images. While some methods have combined these approaches, they often still rely on single-view inputs for training. This paper addresses the gap of effectively utilizing multi-view image data, which is available in many real-world capture systems, to improve the accuracy and robustness of 3D human pose and shape estimation.

Methods

The core of this paper is a collaborative learning framework that tightly integrates a CNN-based regression module with a multi-view optimization module.

  1. CNN Regression: A convolutional neural network (CNN) takes multi-view images as input and regresses the pose and shape parameters of the SMPL model, along with weak perspective camera parameters for each view. This provides an initial estimate.
  2. Multi-View SMPLify (MV-SMPLify): This is an enhanced optimization-based fitting method. Instead of just fitting to 2D joint points from a single view, it simultaneously optimizes the SMPL model's pose and shape parameters across all input views. Crucially, it also optimizes the body orientation for each view, rather than assuming a fixed camera orientation. The CNN's regression output serves as the initialization for this optimization.
  3. Collaborative Training Loop: The framework is self-supervised. The optimized pose and shape parameters from MV-SMPLify are used as ground truth to supervise the training of the CNN. This creates a feedback loop where the CNN provides better initializations for the optimizer, and the optimizer's refined results improve the CNN's learning. This process leverages the inherent relationships between multi-view images to provide richer supervision.

Datasets

The paper uses the following datasets:

  • Human3.6M: A widely used dataset for 3D human pose and shape estimation, captured from four views.
  • MPI-INF-3DHP: Another benchmark dataset, captured from multiple views, used for both training and testing.
  • 3DPW: Used for evaluating performance on outdoor scenarios, providing ground truth 3D pose.

The authors train their model on a combination of Human3.6M and MPI-INF-3DHP, and evaluate on all three.

Key Insights and Ideas

  • Multi-view synergy: The paper demonstrates that explicitly leveraging the geometric constraints and complementary information present in multi-view images significantly reduces ambiguity in 3D pose and shape estimation compared to single-view methods.
  • Self-supervised collaboration: The tight integration of regression (CNN) and optimization (MV-SMPLify) in a self-supervised loop is a key innovation. This allows the model to learn from its own refined outputs, reducing reliance on external ground truth and improving robustness.
  • Optimizing body orientation: Instead of assuming fixed camera orientations, optimizing the body's orientation relative to the camera in each view is shown to be more effective for multi-view fitting.
  • Beyond 2D/3D joint losses: Incorporating losses based on the optimized SMPL mesh and 3D joint points from the optimized model provides stronger supervision for the CNN, leading to better shape recovery.

Key Results

  • Quantitative Improvements: The proposed method achieves state-of-the-art or competitive results across Human3.6M, MPI-INF-3DHP, and 3DPW datasets, outperforming previous single-view and some multi-view approaches in terms of reconstruction error (MPJPE) and other metrics. For instance, on Human3.6M, their method achieved a reconstruction error of 43.8 mm, outperforming SPIN (44.2 mm) and other methods.
  • Qualitative Improvements: Visual results show that the method produces more natural and accurate 3D human body reconstructions, especially in challenging outdoor scenarios (3DPW), where it clearly outperforms SPIN and Liang et al. [23]. The paper highlights that the optimization component helps recover better body shapes, preventing unnatural deformations seen in purely regression-based methods.
  • Ablation Study: An ablation study confirms the benefit of the multi-view SMPLify component, showing that its inclusion leads to improved accuracy and more natural 3D body shapes, particularly on the 3DPW dataset.

Limitations and Future Work

  • Training Data: The method relies on multi-view datasets for training. While it can perform inference on single-view images, its full potential is realized when trained with multi-view data.
  • Computational Cost: While the CNN regression is fast, the optimization step can be time-consuming. The paper mentions using Adam with 0.01 learning rate for 100 iterations for optimization.
  • Future Work: The paper implicitly suggests that extending this framework to handle more complex scenarios, such as dynamic scenes with multiple interacting people or scenes with occlusions, would be valuable future research directions. The current work focuses on single person estimation.

Summary updated

BibTeX
@inproceedings{li2021d,
  author = {Zhongguo Li and Magnus Oskarsson and Anders Heyden},
  title = {{3D Human Pose and Shape Estimation Through Collaborative Learning and Multi-View Model-Fitting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure