CEE-Net: Complementary End-to-End Network for 3D Human Pose Generation and Estimation

Haolun Li, Chi-Man PunAAAI 2023 · DOI

Introduction

3D Human Pose Estimation (HPE)—the task of predicting 3D joint coordinates from 2D inputs—is a cornerstone of computer vision with applications ranging from sports analytics to human-computer interaction. While current models achieve high accuracy on standard benchmarks like Human3.6M, they often struggle with "cross-dataset" generalization. When faced with unseen actors, rare actions, or unusual camera angles, performance typically degrades significantly. CEE-Net (Complementary End-to-End Network) addresses this by introducing a co-training framework that harmonizes data generation and robust estimation.

The Challenge: Data Scarcity and Diversity

The primary bottleneck in 3D HPE is the lack of diverse training data. Existing datasets are limited in the variety of actions and actors they contain. While previous data augmentation techniques like PoseAug successfully varied camera viewpoints (RT) and bone lengths (BL), they struggled to diversify bone angles (BA)—the actual poses themselves. Generating complex or "rare" poses often leads to training instability, as standard estimators cannot effectively learn from highly irregular data, a phenomenon known as training collapse.

Methodology: The Complementary Framework

CEE-Net overcomes these hurdles through a "complementary" design where the generator and estimator are specifically engineered to support one another.

The CEE-Net architecture, featuring the CEE-Generator and the BPMixer-based CEE-Estimator.

1. CEE-Generator

To move beyond simple geometric transformations, the CEE-Generator employs an adaptive interpolation sub-net. This component learns to interpolate between known actions from existing datasets, creating new, anatomically valid intermediate poses. To ensure these generated poses remain physically plausible, the authors incorporate a Joint-Angle Loss derived from real-world anatomical limits of professional athletes.

2. CEE-Estimator (BPMixer)

To handle the increased complexity of the generated poses, the authors developed BPMixer (Body-Parts Mixer). This all-MLP architecture decomposes the human body into five distinct groups: the torso and four limbs.

  • Internal-mix MLP: Captures spatial dependencies within a specific limb.
  • External-mix MLP: Facilitates information exchange between different body parts to ensure global structural coherence. This grouping strategy makes the estimator significantly more robust, allowing it to utilize "hard" poses created by the generator without destabilizing the training process.

Key Insights

  • The Complementary Loop: The synergy between a more capable generator and a more robust estimator is the core innovation. BPMixer's stability allows the generator to push the boundaries of bone-angle diversity further than previously possible.
  • Bone Angle is the Frontier: Ablation studies confirm that while camera viewpoint and bone length variations are beneficial, diversifying bone angles provides the most significant gains for cross-dataset generalization.
  • Anatomical Priors: Using explicit joint-angle limits ensures that the generated data remains grounded in human biology, preventing the creation of unrealistic "noodle-like" skeletons.

Experimental Results

CEE-Net achieves state-of-the-art performance on major cross-dataset benchmarks, demonstrating its superior ability to generalize to unseen environments and "in-the-wild" scenarios.

Comparison of CEE-Net against state-of-the-art methods on 3DHP and 3DPW datasets.
  • MPI-INF-3DHP (3DHP): CEE-Net achieved an MPJPE of 69.7mm, outperforming the previous state-of-the-art DH-AUG (71.2mm) and drastically improving over the non-augmented baseline (86.6mm).
  • 3DPW Dataset: On this challenging "in-the-wild" dataset, CEE-Net reached a PA-MPJPE of 76.8mm, a notable improvement over PoseAug (81.6mm) and DH-AUG (79.3mm).
  • Data Efficiency: The framework remains highly effective even when trained on limited subsets of data, highlighting its efficiency in leveraging available information.

Conclusion and Future Directions

CEE-Net represents a significant step forward in making 3D HPE models more reliable for real-world use. By focusing on bone-angle diversity and structural robustness, it breaks through the generalization plateau of previous methods. However, the model still depends on the quality of initial 2D pose detections. Future research will likely focus on making these systems more resilient to severe occlusions and motion blur, perhaps by integrating 2D detection more deeply into the end-to-end training loop.

Summary updated

BibTeX
@inproceedings{li2023cee,
  author = {Haolun Li and Chi-Man Pun},
  title = {{CEE-Net: Complementary End-to-End Network for 3D Human Pose Generation and Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25214},
}
Figure