TL;DR
This paper introduces LInKs, a novel unsupervised method for 3D human pose estimation from 2D keypoints that excels at handling occlusions. By lifting independent keypoint segments to 3D before filling in missing information, LInKs achieves significantly higher accuracy than previous methods, particularly in challenging real-world scenarios.
What Problem Does This Paper Address?
Estimating a person's 3D pose from a single 2D image is a notoriously difficult problem. This is because multiple 3D poses can project to the same 2D pose, making it an ill-posed inverse problem. Traditional solutions often rely on multiple camera views or depth sensors, which aren't always practical. While unsupervised learning methods have made strides, they typically struggle with occlusions – situations where parts of the body are hidden from view. Existing methods often fail completely if even a single keypoint is missing, and they can also learn spurious long-range correlations between body parts, leading to inaccurate estimations. This paper tackles the critical challenge of robust 3D pose estimation in the presence of occlusions, aiming to improve accuracy and reliability in real-world applications.
Methods
LInKs employs a unique "lift-then-fill" two-stage approach.
- Independent Lifting Networks: Instead of lifting the entire 2D skeleton at once, LInKs breaks the 2D pose into segments (torso, legs, left/right limbs) and lifts each segment to a partial 3D pose independently. This is crucial for occlusion handling, as missing keypoints in one segment don't prevent other segments from being lifted. This also helps reduce the learning of anatomically unassociated long-range correlations.
- Occlusion Handling Network: A dedicated network then takes the partial 3D pose and fills in the missing keypoints, reconstructing a complete 3D pose. This occlusion handling is performed in 3D space, which the authors argue is more robust than trying to infer missing 2D keypoints first.
Key innovations include:
- Normalizing Flows with Generative Sampling: To address the limited data in unsupervised learning, the authors leverage normalizing flows. They introduce a novel generative sampling strategy that augments training data by generating realistic 2D poses, improving the flow's ability to learn meaningful representations and avoiding issues with random sampling that can lead to unnatural poses.
- Rotational Consistency and Re-projection Likelihood: The model enforces self-consistency by virtually rotating and re-projecting 3D poses back into 2D, ensuring that the learned 3D poses are consistent with the input 2D data across different viewpoints.
- Independent Lifting Loss: Each lifting network incurs its own loss based on the keypoints it's responsible for, further promoting robustness.
Datasets
The paper evaluates its approach on two widely used datasets:
- Human3.6M: A large-scale dataset with motion capture data and videos from four viewpoints.
- MPI-INF-3DHP: A markerless motion capture dataset.
The authors also conducted an additional study on simulated occlusion scenarios using the Human3.6M dataset.
Key Insights and Ideas
- Lifting in 3D is Superior to 2D for Occlusions: The core insight is that handling occlusions in 3D space, after lifting partial poses, leads to significantly better results than trying to infer missing 2D keypoints first and then lifting.
- Independent Lifting Reduces Complexity and Errors: Breaking down the pose into independent segments for lifting not only makes the problem more manageable but also inherently reduces the propagation of errors from occluded parts.
- Generative Sampling Enhances Normalizing Flows: The custom sampling strategy for normalizing flows is critical for generating realistic training data and improving the model's overall performance in an unsupervised setting.
- Anatomical Constraints Matter: The "lift-then-fill" approach in 3D space is better at respecting human anatomical constraints compared to filling in 2D and then lifting, which can lead to unnatural poses.
Key Results
- Human3.6M: LInKs achieves a 7.9% improvement in PA-MPJPE over the previous unsupervised state-of-the-art [25] and a 5% improvement in N-MPJPE. It also outperforms other unsupervised methods.
- MPI-INF-3DHP: LInKs demonstrates superior performance compared to previous unsupervised and weakly-supervised approaches, achieving a PA-MPJPE of 54.0mm.
- Occlusion Handling: In simulated occlusion scenarios on Human3.6M, LInKs (O3D) significantly outperforms its 2D-filling counterpart (O2D), achieving lower PA-MPJPE and N-MPJPE across various occlusion types (e.g., 52.1mm PA-MPJPE for Left Arm occlusion vs. 61.4mm for O2D).
- Qualitative Results: Figures in the paper showcase impressive qualitative results, demonstrating the model's ability to reconstruct complete and natural-looking 3D poses even with significant occlusions.
Limitations and Future Work
- Across-Body Occlusions: The current approach struggles with occlusions that span across the body (e.g., left wrist and right ankle simultaneously missing).
- Specific Errors: The leg occlusion networks sometimes produce errors, such as incorrectly predicting crouching or lunging poses.
- Future Work: The authors plan to address these limitations by developing more robust lift-then-fill approaches and investigating adaptive network structures that can handle varying input dimensions for partial 3D poses. They also aim to improve handling of across-body occlusions.
Summary updated