Motion Projection Consistency-Based 3-D Human Pose Estimation With Virtual Bones From Monocular Videos

Guangming Wang, Honghao Zeng, Ziliang Wang, Zhe Liu, Hesheng Wang

TL;DR

This paper introduces "virtual bones" to improve 3D human pose estimation from monocular videos, addressing cumulative errors in traditional methods. By adding imaginary bones between non-adjacent joints, they create new loop constraints that enhance joint position accuracy. They also propose a novel "projection consistency loss" to smooth pose predictions across video frames.

What Problem Does This Paper Address?

Estimating 3D human pose from a single camera (monocular video) is challenging due to the inherent depth ambiguity. While many methods first detect 2D keypoints and then infer 3D pose, a common issue is the accumulation of errors, especially when estimating bone lengths and directions sequentially. This can lead to inaccurate and unstable pose predictions, particularly for joints at the extremities of the body. The paper aims to mitigate these cumulative errors and improve the overall accuracy and smoothness of 3D pose estimation from video.

Methods

The core innovation lies in the introduction of virtual bones. These are imaginary connections between non-adjacent joints (e.g., pelvis to right-elbow) that, while not physically present, provide additional constraints for the pose estimation network. The proposed network simultaneously predicts the lengths of both real (adjacent) and virtual bones.

  • Virtual Bones for Loop Constraints: By predicting virtual bone lengths, the network can form "loops" of constraints. For instance, a virtual bone from the pelvis to the right elbow, combined with real bones, can help constrain the position of the right elbow more robustly than relying solely on sequential bone estimations. This helps to reduce cumulative errors.
  • Attention Mechanism for Real Bone Lengths: To account for the invariance of real bone lengths, the paper uses an attention mechanism to weigh the predicted real bone lengths from randomly sampled frames, aiming to capture the most representative length for the current frame. Virtual bone lengths are predicted directly from the current frame.
  • Projection Consistency Loss: A novel loss function is introduced to enforce consistency between the 2D projection of the estimated 3D joint movements and the actual 2D joint movements observed in the video. This loss helps to smooth out pose fluctuations across consecutive frames without requiring 3D ground truth for every frame. It leverages the pinhole camera model to project estimated 3D points back to 2D and compares their displacement to the ground truth 2D displacement.
  • Bone Direction Prediction: The paper utilizes a temporal fully-convolutional network, similar to prior work, to predict the direction of bones.

Datasets

The primary dataset used for training and evaluation is Human3.6M. This dataset provides annotated 2D and 3D joint positions from video sequences captured by multiple camera views across various human activities. The authors use a standard split, with data from 5 subjects for training and 2 subjects for testing.

Key Insights and Ideas

  • Imaginary Constraints for Real Accuracy: The idea that "imaginary" bones can provide real-world improvements in pose estimation is a key insight. Virtual bones act as a form of regularization, preventing the network from relying too heavily on potentially erroneous sequential estimations.
  • Loop Constraints Mitigate Cumulative Error: By creating closed loops with virtual bones, the system can better constrain joint positions, directly addressing the problem of error accumulation that plagues sequential bone-based methods.
  • Temporal Consistency via Projection: The projection consistency loss is a clever way to enforce temporal smoothness. By ensuring that the 2D projection of estimated 3D motion aligns with observed 2D motion, the method implicitly constrains the 3D pose to follow natural human movement patterns.
  • Attention for Robust Bone Lengths: Using an attention mechanism to aggregate real bone lengths from multiple sampled frames adds robustness, especially when dealing with variations or potential noise in individual frames.

Key Results

The proposed method demonstrates good performance on the Human3.6M dataset. Ablation studies show that:

  • Adding virtual bones generally improves accuracy across various metrics (Protocol 1, 2, 3, and Velocity). The best performance is achieved with a combination of virtual bones (e.g., 10, 13, or 23 virtual bones).
  • The projection consistency loss also contributes to improved accuracy, particularly for the 9-frame receptive field model, and helps to smooth pose predictions.
  • The paper reports competitive results compared to state-of-the-art methods, especially when using ground truth 2D inputs. For example, their 243-frame model achieves a Protocol 1 MPJPE of 32.5, outperforming Chen et al. [14] (34.0) on this metric.

Limitations and Future Work

  • Parameter Sensitivity: The authors note that the method is sensitive to the choice of hyperparameters, requiring careful tuning as demonstrated in their parameter sensitivity tests.
  • Computational Cost: While they offer a lightweight 9-frame model, the 243-frame model, while more accurate, exceeds real-time requirements (1.19s for 25 frames).
  • Future Work: The authors suggest exploring graph networks composed of real and virtual bones as a future direction, indicating a potential for further advancements in leveraging the structural relationships between joints.

Summary updated

Abstract

Real-time 3-D human pose estimation is crucial for human–computer interaction. It is cheap and practical to estimate 3-D human pose only from monocular video. However, the recent bone-splicing-based 3-D human pose estimation method brings about the problem of cumulative error. In this article, the concept of virtual bones is proposed to solve such a challenge. The virtual bones are imaginary bones between nonadjacent joints. They do not exist in reality, but they bring new loop constraints for the estimation of 3-D human joints. The proposed network in this article predicts real bones and virtual bones, simultaneously. The final length of real bones is constrained and learned by the loop constructed by the predicted real bones and virtual bones. Besides, the motion constraints of joints in consecutive frames are considered. The consistency between the 2-D projected position displacement predicted by the network and the captured real 2-D displacement by the camera is proposed as a new projection consistency loss for the learning of the 3-D human pose. The experiments on the Human3.6M data set demonstrate the good performance of the proposed method. Ablation studies demonstrate the effectiveness of the proposed interframe projection consistency constraints and intraframe loop constraints.
BibTeX
@article{wang2023motion,
  author = {Guangming Wang and Honghao Zeng and Ziliang Wang and Zhe Liu and Hesheng Wang},
  title = {{Motion Projection Consistency-Based 3-D Human Pose Estimation With Virtual Bones From Monocular Videos}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=vwJIsLy88y},
}
Figure