TL;DR
This paper introduces PoseLifter, a novel neural network that can accurately estimate a person's absolute 3D pose from a single, potentially noisy, 2D human pose. By leveraging decomposed 2D pose information (location and scale) and a new "canonical root depth" representation, PoseLifter achieves state-of-the-art performance in both 2D-to-3D pose lifting and full 3D human pose estimation from images.
What Problem Does This Paper Address?
Estimating a 3D human pose from a single 2D image is a fundamentally ill-posed problem. While existing methods can infer a relative 3D pose (joints positioned relative to a root joint), determining the absolute 3D position of the entire skeleton in camera coordinates, especially the depth of the root joint, is challenging. This ambiguity arises from factors like the varying size of humans in images and the unknown focal length of the camera. The paper aims to solve this by proposing a method that directly outputs an absolute 3D pose, including the root's precise location in 3D space.
Methods
The core contribution is the PoseLifter network. It takes a 2D human pose as input and outputs an absolute 3D pose. Key methodological innovations include:
- Normalization Layer: This initial layer processes the input 2D pose by:
- Translating the 2D joint coordinates to be independent of the camera's principal point.
- Normalizing the 2D pose to have zero mean and unit variance.
- Crucially, it also extracts and concatenates the 2D location and scale of the subject as intermediate features. These are vital for estimating the absolute root position.
- Canonical Root Depth: Instead of directly predicting the absolute depth of the root joint (which is highly sensitive to camera focal length), PoseLifter predicts a "canonical root depth." This depth is normalized by the focal length, making it independent of this variable and thus more robust. The actual absolute depth can be recovered if the focal length is known.
- Cascade Approach: For full 3D human pose estimation from an RGB image, PoseLifter is used in a cascade with a 2D pose estimator. The 2D pose is first detected, then fed into PoseLifter.
- Synthesized Error for Training: To make the PoseLifter robust to errors from the 2D pose estimator, the authors propose synthesizing realistic 2D pose errors. They model these errors using a mixture of Gaussian and uniform distributions, which are then used to generate training data for PoseLifter.
Datasets
- Human3.6M: Used as the primary dataset for training and evaluating the proposed method. It provides large-scale 3D human pose data from multiple cameras.
- MPI-INF-3DHP: Also used for evaluation, this dataset features a commercial motion capture system and has significantly different focal length parameters across its training and testing sets, making it a good testbed for the canonical depth approach.
- MPII: Used for training the 2D pose estimator.
Key Insights and Ideas
- 2D Location and Scale are Crucial for Absolute Pose: The paper empirically demonstrates that simply using normalized 2D joint coordinates is insufficient for absolute 3D pose estimation. Incorporating the 2D location and scale of the subject provides essential cues for inferring the root's 3D position.
- Canonical Root Depth Solves Focal Length Ambiguity: Directly predicting absolute depth is problematic due to unknown camera focal lengths. The "canonical root depth" representation effectively decouples this dependency, allowing for more generalizable pose lifting.
- Realistic Error Synthesis is Key for Robustness: 2D pose estimation errors are inevitable. By carefully modeling and synthesizing these errors, the PoseLifter can be trained to be robust to imperfect 2D inputs, leading to better overall 3D pose estimation performance.
- Cascade Approach with Error Modeling Outperforms Direct Methods: The proposed cascade approach, enhanced by realistic error synthesis, achieves competitive or superior results compared to many direct 3D pose estimation methods, especially those that require ground truth depth.
Key Results
- 2D-to-3D Pose Lifting: PoseLifter, using location and scale information, achieved an MRPE (Mean Root Position Error) of 98.84mm on Human3.6M, significantly outperforming methods that only estimate relative poses. It also reduced the root-relative MPJPE from 44.86mm to 38.38mm.
- 3D Human Pose Estimation (Cascade): The cascade approach with a mixture model for error synthesis achieved state-of-the-art results on Human3.6M, outperforming other cascade methods and showing competitive performance against direct methods. For example, on Protocol 1, it achieved an MPJPE of 38.38mm and PA-MPJPE of 29.91mm.
- MPI-INF-3DHP: The method also performed well on this dataset, demonstrating its generalization capabilities, especially with the canonical depth representation which significantly reduced MRPE.
Limitations and Future Work
- Reliance on 2D Pose Estimator: The performance of the cascade approach is inherently limited by the accuracy of the initial 2D pose estimator.
- Image Features vs. 2D Poses: The paper notes that some direct methods achieve better results by utilizing image features, whereas their approach relies solely on 2D poses. This suggests that integrating image features could further improve performance.
- Focal Length Information: While the canonical depth is independent of focal length, recovering the absolute root depth still requires knowing the camera's focal length.
- Future Work: The authors suggest that their method can be used in conjunction with object detectors for challenging "in-the-wild" images, indicating potential for broader applications. The success of their error modeling also points towards further research in robust training strategies for pose estimation tasks.
Summary updated