Pose ResNet: 3D Human Pose Estimation Based on Self-Supervision

Wenxia Bao, Zhongyu Ma, Dong Liang, Xianjun Yang, Tao Niu2023 · DOI

TL;DR

This paper introduces Pose ResNet, a novel self-supervised deep learning model for 3D human pose estimation from single 2D images. By leveraging epipolar geometry to generate pseudo-labels, it bypasses the need for expensive 3D ground truth annotations, achieving competitive accuracy and paving the way for more accessible 3D pose estimation.

What Problem Does This Paper Address?

Accurately estimating the 3D pose of a human from images is crucial for applications like human-computer interaction, motion analysis, and autonomous driving. However, a major bottleneck is the scarcity and high cost of acquiring 3D ground truth data, typically obtained through motion capture systems. Existing methods often rely on these expensive datasets, limiting their applicability in real-world, unconstrained environments. This paper tackles this challenge by proposing a self-supervised approach that learns 3D pose estimation directly from 2D images without requiring explicit 3D annotations.

Methods

The core of the proposed method is the Pose ResNet architecture, built upon a ResNet50 backbone for feature extraction. It incorporates two key modules:

  • Convolutional Block Attention Module (CBAM): This module enhances feature representation by selectively focusing on important pixels and suppressing irrelevant background information, improving the accuracy of joint detection.
  • Waterfall Atrous Spatial Pooling (WASP) Module: This module expands the receptive field and captures multi-scale contextual information, crucial for handling occlusions and inferring the pose of occluded joints.

The self-supervision is achieved through a clever two-branch network. The upper branch estimates a 2D pose, which is then used by an epipolar geometry module to generate synthetic 3D "pseudo-labels." These pseudo-labels are then used to train the lower branch, a 3D Pose CNN, effectively creating a self-supervised learning loop. The model also benefits from transfer learning, pre-training on a 2D dataset (MPII) before undergoing self-supervised training on a 3D dataset (Human3.6M). Additionally, synthetic occlusion is applied to the training data to improve the model's robustness and generalization capabilities.

Datasets

  • MPII: Used for pre-training the 2D pose estimation component. This dataset contains approximately 25,000 images of human behaviors captured in outdoor environments.
  • Human3.6M: Used for self-supervised training and evaluation. This is a large-scale dataset with 3.6 million RGB images captured in indoor environments using motion capture systems, providing 3D ground truth annotations. The paper utilizes a subset of this data for training and testing.

Key Insights and Ideas

  • Self-Supervision via Epipolar Geometry: The most significant contribution is the use of epipolar geometry to generate 3D pseudo-labels from multi-view 2D poses. This elegantly bypasses the need for expensive 3D ground truth, making 3D pose estimation more accessible.
  • Bridging 2D and 3D: The transfer learning approach, moving from 2D pose estimation on MPII to 3D pose estimation on Human3.6M, effectively leverages existing knowledge and improves the accuracy of the generated 3D labels.
  • Robustness through Synthetic Occlusion: Augmenting training data with synthetic occlusions significantly improves the model's ability to handle real-world scenarios where body parts are often partially hidden.
  • Attention and Multi-Scale Features: The integration of CBAM and WASP modules demonstrates the effectiveness of attention mechanisms and multi-scale feature extraction in improving pose estimation accuracy, especially in complex scenes.

Key Results

The proposed self-supervised Pose ResNet achieved a mean per joint position error (MPJPE) of 74.6 mm on the Human3.6M dataset. This result is competitive, especially considering it was achieved without relying on 3D ground truth labels. The paper also shows that:

  • Transfer learning alone reduced MPJPE by 19 mm.
  • Synthetic occlusion further reduced MPJPE by 7.6 mm.
  • CBAM and WASP modules contributed an additional reduction of 1.5 mm.
  • Ablation studies indicated that deeper ResNet backbones (like ResNet101) offered slightly better 2D pre-training accuracy (PCK) but were not chosen due to computational cost, with ResNet50 being a good balance.

Limitations and Future Work

  • Small Target Problem: The authors acknowledge that the model struggles with "small target problems," where the person occupies a small area in the image or is not centered. This limits its applicability in natural scenes with distant subjects.
  • Future Work: The authors suggest further optimization for the small target problem to improve performance in natural scenes. They also imply that exploring more advanced architectures or training strategies could further enhance accuracy.

Summary updated

BibTeX
@misc{bao2023pose,
  author = {Wenxia Bao and Zhongyu Ma and Dong Liang and Xianjun Yang and Tao Niu},
  title = {{Pose ResNet: 3D Human Pose Estimation Based on Self-Supervision}},
  year = {2023},
  doi = {10.3390/s23063057},
}
Figure