TL;DR
This paper introduces TIPSy-GAN, a novel unsupervised method for 2D to 3D human pose estimation. By treating the human skeleton as independent parts (torso and legs) during training, TIPSy-GAN significantly improves accuracy and stability, outperforming previous unsupervised methods and even rivaling supervised approaches.
What Problem Does This Paper Address?
Estimating a 3D human pose from a single 2D image is a fundamentally ill-posed problem. Many existing methods, especially unsupervised ones that learn from 2D data alone, rely on adversarial training. However, these methods often assume the entire human kinematic skeleton is a single, spatially interdependent structure. The authors hypothesize that this assumption leads to learned biases where the 3D position of one body part is incorrectly correlated with the 2D positions of distant parts. This can hinder accuracy and stability, particularly in truly unsupervised settings where ground truth 3D data is unavailable for training.
Methods
TIPSy-GAN tackles this by breaking down the 2D pose into spatially independent components: the torso and the legs. Instead of training a single generator for the entire pose, they train two separate generators, one for the torso and one for the legs, using an adversarial approach.
Key innovations include:
- Independent Part Generators: Training separate generators for the torso and legs to avoid learning spurious correlations between distant body parts.
- Enhanced Self-Consistency: Introducing new consistency constraints during training, specifically 90-degree rotations and reprojection consistency, to improve the learning process and reduce evaluation error.
- Knowledge Distillation: A final, end-to-end TIPSy model is trained by distilling the knowledge from the specialized torso and leg generators. This allows for efficient prediction of the full 3D pose while retaining the benefits of the independent part training.
- Adversarial Stability: The paper demonstrates that training two independent generators leads to more stable adversarial learning compared to a single generator, which can collapse.
Datasets
- Human3.6M: Used for training and evaluation. This is a large-scale dataset containing 3.6 million 3D human poses from motion capture data.
- MPI-INF-3DHP: Used for evaluating the model's generalizability to unseen poses. This is a markerless MoCap dataset with 8 actors performing various activities.
Key Insights and Ideas
- Decomposition is Key: The core idea that treating the human skeleton as independent parts (torso and legs) during unsupervised training is crucial for improving accuracy and stability. This challenges the common assumption of a single, interdependent structure.
- Self-Consistency is Paramount: The paper strongly emphasizes that improving self-consistency constraints during adversarial training is a primary driver for reducing evaluation error.
- Unsupervised Stability: Training multiple generators (torso and legs) adversarially offers significantly better stability than a single generator, preventing collapse and leading to more reliable learning.
- Knowledge Distillation for Efficiency: Distilling knowledge from specialized generators into a single end-to-end model provides a practical way to achieve the benefits of independent part learning without the computational overhead of multiple models during inference.
Key Results
- 17% Reduction in MPJPE: TIPSy-GAN achieved a 17% decrease in Mean Per Joint Position Error (MPJPE) on the Human3.6M dataset compared to a baseline solo generator.
- Competitive Performance: TIPSy-GAN performs strongly against supervised and weakly-supervised approaches on both Human3.6M and MPI-INF-3DHP datasets.
- Improved Generalization: The model trained on Human3.6M showed high AUC performance on the MPI-INF-3DHP dataset, demonstrating good generalizability to unseen poses.
- Adversarial Stability: Figure 3 shows that the leg and torso generators exhibit lower and more stable error during adversarial training compared to solo generators.
Limitations and Future Work
- Unintuitive Correlations: While TIPSy-GAN reduces overall error and some unintuitive correlations, the authors note that further investigation is needed to fully understand how changing one 2D keypoint affects 3D predictions across different models. They observed some unexpected correlations even in TIPSy, though generally fewer than in the solo generator.
- Computational Cost of Derivatives: The paper mentions that implementing a theoretical consistency constraint involving derivatives is computationally inefficient and requires removing batch normalization layers, which impacts stability. They are exploring alternatives like layer normalization.
- Further Investigation of Divergence: The authors are interested in preempting the observed divergence in GAN training by analyzing changes in loss functions.
Summary updated