Abstract
The task of three-dimensional (3D) human pose estimation from a single image can be divided into two parts: (1) Two-dimensional (2D) human joint detection from the image and (2) estimating a 3D pose from the 2D joints. Herein, we focus on the second part, i.e., a 3D pose estimation from 2D joint locations. The problem with existing methods is that they require either (1) a 3D pose dataset or (2) 2D joint locations in consecutive frames taken from a video sequence. We aim to solve these problems. For the first time, we propose a method that learns a 3D human pose without any 3D datasets. Our method can predict a 3D pose from 2D joint locations in a single image. Our system is based on the generative adversarial networks, and the networks are trained in an unsupervised manner. Our primary idea is that, if the network can predict a 3D human pose correctly, the 3D pose that is projected onto a 2D plane should not collapse even if it is rotated perpendicularly. We evaluated the performance of our method using Human3.6M and the MPII dataset and showed that our network can predict a 3D pose well even if the 3D dataset is not available during training.
Introduction
Estimating a 3D human pose—the $(x, y, z)$ coordinates of joints—from a single 2D image is a classic ill-posed problem in computer vision. Because depth information is lost during projection, a single 2D point can correspond to an infinite number of 3D locations. Traditionally, researchers solve this by "lifting" 2D joints to 3D using massive datasets captured in expensive motion-capture (MoCap) labs. However, these datasets are limited in variety and often fail to generalize to "in-the-wild" scenarios like outdoor sports. While some methods use video sequences to infer depth through motion, they cannot operate on single snapshots. This paper introduces the first unsupervised framework to learn 3D human poses from single 2D images without requiring any 3D training data.
Methodology: Adversarial Learning with Geometric Constraints
The proposed framework utilizes a Generative Adversarial Network (GAN) to estimate depth from 2D joint locations. The architecture is intentionally simple, employing linear layers with skip connections to demonstrate that the strength of the approach lies in the training logic rather than model complexity.
The GAN Architecture
The system consists of a generator and a discriminator. The generator takes 2D joint coordinates $(x, y)$ and predicts the missing depth components $(z)$. To train this without 3D supervision, the authors introduce a "rotation-projection" loop. The generated 3D pose is rotated by a random angle $\theta$ around the vertical axis and then projected back to a 2D plane.
Geometric Intuition
The core insight is that a valid 3D human pose should look like a realistic human pose from any horizontal viewing angle. If the generator predicts an incorrect depth, the resulting 3D structure will "collapse" or appear anatomically impossible when rotated and projected back to 2D. The discriminator is trained to distinguish between "real" 2D poses from a dataset and these "fake" projected poses. To fool the discriminator, the generator must learn to produce anatomically plausible 3D structures.
To resolve depth ambiguity (e.g., front-back mirroring), a heuristic angle loss is added. This ensures basic anatomical consistency, such as the right shoulder remaining on the right side of the head relative to the face orientation.
Experiments and Results
The model was evaluated on the Human3.6M dataset and the MPII Human Pose Dataset.
- Human3.6M: Although this dataset contains 3D ground truth, it was only used for evaluation. The model achieved an average error of 130.9mm using ground-truth 2D joints. While higher than supervised methods (which average ~45mm), this is a landmark result for a system that never saw a 3D coordinate during training.
- MPII: This dataset contains real-world images without 3D labels. The model successfully reconstructed natural-looking 3D poses for complex activities like cricket or horse riding, demonstrating strong generalization capabilities beyond lab environments.
Conclusion
This research demonstrates that 3D human anatomy can be learned from 2D data alone by leveraging adversarial training and geometric constraints. By removing the dependency on expensive MoCap data, this approach enables 3D pose estimation for joints and activities that are difficult to capture in a lab. Future work will focus on incorporating camera perspective parameters to further improve accuracy and handle more varied camera angles.
Summary updated