Decanus to Legatus: Synthetic Training for 2D-3D Human Pose Lifting

Yue Zhu, David Picard

Introduction

Estimating 3D human pose from a single 2D image remains a fundamental challenge in computer vision. This task is inherently "ill-posed," as a single 2D projection can correspond to multiple valid 3D configurations—for instance, an arm reaching toward the camera may appear identical to one reaching away. Traditionally, solving this ambiguity has required deep learning models trained on massive, high-fidelity 3D ground-truth datasets.

However, acquiring 3D data is a significant bottleneck. It typically requires specialized motion-capture (Mocap) laboratories, which are expensive and limit the variety of actions and environments captured. This creates a "domain gap" when models are applied to "in-the-wild" images. This research introduces a framework called Decanus to Legatus, which explores a provocative question: Can we train a robust 3D pose estimator without using any real 3D training data at all?

Methodology

The authors propose a paradigm shift from data collection to data evolution. The framework moves from a small set of initial "seeds" (Decanus) to an infinite, evolving synthetic population (Legatus) used for training.

1. The Markov Tree Model

To ensure biological plausibility, the authors model the human skeleton as a hierarchical tree rather than a collection of independent points. They utilize a local spherical coordinate system (radius, polar angle, and azimuthal angle) centered on parent joints. This representation naturally enforces fixed bone lengths and allows the model to focus on learning valid angular ranges, which is more efficient than navigating global Cartesian coordinates.

2. Decanus: The Seed Poses

The process begins with only 10 handcrafted 3D poses. These seeds are generated semi-automatically by selecting high-variance 2D poses from existing real-world datasets and manually "lifting" them into 3D by estimating relative depths. These 10 poses serve as the genetic foundation for the entire synthetic dataset.

3. Distribution Diffusion

To prevent the neural network from overfitting to the limited seed set, the authors implement a distribution diffusion process. Inspired by the heat diffusion equation, they apply a Laplacian operator to the probability distributions of the joint angles. Over the course of training, the distribution of poses "spreads out," gradually introducing more diverse and complex configurations. This serves as a form of curriculum learning: the model masters simple, seed-like poses first before progressively tackling the more difficult, diffused variations.

4. Scaleless Training

Because 2D images lack absolute depth information, the model is trained using a scaleless orthogonal projection. A multiview consistency loss is applied, ensuring that the predicted 3D pose remains structurally coherent when viewed from multiple virtual camera angles.

Experimental Results

The model was trained exclusively on synthetic data generated by the Legatus algorithm and evaluated on standard benchmarks to test its zero-shot capabilities.

  • Quantitative Performance: On the Human3.6M dataset, the model achieved a Mean Per Joint Position Error (MPJPE) of 60.8mm. This is a significant improvement over previous synthetic-only methods, which often exceed 100mm in error. Furthermore, this performance is competitive with "weakly supervised" methods that utilize real 2D data during training.
  • Generalization: When tested on "in-the-wild" datasets like COCO, MPII, and 3DPW, the model successfully lifted 2D skeletons into plausible 3D shapes. This demonstrates that the model learned the underlying manifold of human movement rather than just memorizing specific training examples.

Key Insights

This research highlights that structure often outweighs statistics. One does not necessarily need a million real Mocap frames to understand human kinematics; a few well-chosen structural seeds combined with a smart probabilistic evolution model can capture the essence of biologically plausible poses. The use of the heat equation to evolve the pose distribution ensures a "natural" expansion of the search space, allowing the model to discover the manifold of human movement autonomously.

Limitations and Future Work

Despite its success, the framework has areas for improvement:

  • Generator Bias: The authors observed occasional "hallucinations," such as legs bending backward in standing poses, likely due to biases in the initial seed distributions or the diffusion process.
  • Manual Seeding: While 10 poses is a minimal requirement, the process still involves a semi-automatic manual step. Future iterations could focus on fully automating seed selection from 2D data.
  • Robustness: The model’s performance is sensitive to missing keypoints. Improving robustness to occlusions—a common occurrence in real-world imagery—remains a priority for future development.

Conclusion

Decanus to Legatus demonstrates that synthetic data, when structured correctly and evolved through principled probabilistic methods, can rival expensive real-world data for complex tasks like 3D pose lifting. By bridging the gap between handcrafted constraints and deep learning, this approach offers a scalable path forward for 3D human sensing without the constraints of the Mocap lab.

Summary updated

BibTeX
@inproceedings{zhu2022decanus,
  author = {Yue Zhu and David Picard},
  title = {{Decanus to Legatus: Synthetic Training for 2D-3D Human Pose Lifting}},
  booktitle = {ACCV},
  year = {2022},
  url = {https://openreview.net/forum?id=xi0w8wBRJpo},
}
Figure