Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose Estimation

Chenxin Xu, Siheng Chen, Maosen Li, Ya Zhang

Abstract

Estimating 3D human poses from 2D landmarks without 3D supervision is a challenging, ill-posed problem due to depth ambiguity. We propose the Invariant Teacher and Equivariant Student (ITES) framework to address the "regularization-flexibility dilemma." ITES utilizes a heavily regularized, rotation-invariant teacher network to establish physically plausible 3D pose priors and a flexible, rotation-equivariant student network to refine joint depth. Our method achieves state-of-the-art unsupervised performance on the Human3.6M dataset, reducing the Procrustes-aligned mean per-joint position error (P-MPJPE) by 11.4% and outperforming several weakly-supervised approaches.

Introduction

3D human pose estimation is a cornerstone for applications in animation, sports analytics, and human-computer interaction. While supervised methods yield high accuracy, they rely on massive datasets of paired 2D-3D annotations, which are difficult to acquire. Unsupervised learning offers a scalable alternative but suffers from the inherent ambiguity of perspective projection—where multiple 3D configurations can map to the same 2D pose.

Existing unsupervised methods often rely on strong priors to ensure physical plausibility. However, this creates a trade-off: excessive regularization leads to underfitting (inability to capture diverse poses), while insufficient regularization leads to overfitting (physically impossible skeletons). ITES resolves this by decoupling the estimation process into two specialized components.

Conceptual comparison of (a) rotation-invariant estimation in a canonical view and (b) rotation-equivariant estimation in the camera view.

Methodology

The ITES framework consists of a two-stage Teacher-Student distillation process designed to balance structural constraints with representational flexibility.

The Invariant Teacher Network

The teacher network focuses on strict regularization. It employs a "pose-dictionary" approach, representing any 3D pose as a linear combination of learned "pose atoms." This constraint ensures that the output remains within the space of physically plausible human shapes. To resolve viewpoint ambiguity, the teacher is trained with a cycle-consistent loss that enforces 3D rotation-invariance. Regardless of the input 2D view, the teacher is forced to predict the same canonical 3D pose.

The Equivariant Student Network

The student network provides the necessary flexibility. It utilizes an Adaptive Graph Convolutional Network (AGCN) that models both physical bone connections and non-physical joint correlations (e.g., hand-to-foot coordination). Unlike the teacher, the student is trained to be 3D rotation-equivariant, meaning its 3D output should rotate in accordance with changes in the input 2D view. The student learns via knowledge distillation from the teacher, refining the coarse canonical estimates into view-specific 3D poses.

Overview of the ITES framework, featuring the teacher-student distillation process and cycle-consistent losses.

Experiments and Results

The model was primarily trained and evaluated on the Human3.6M dataset. To test generalization, the authors evaluated the model on the MPI-INF-3DHP dataset without further fine-tuning.

Quantitative Performance

ITES achieved a new state-of-the-art for unsupervised 3D pose estimation. On Human3.6M, the P-MPJPE was reduced from 58.0mm to 51.9mm, an 11.4% improvement. Notably, ITES outperformed weakly-supervised methods like RepNet and AIGN, which utilize additional information such as unpaired 3D poses or multi-view data during training.

Ablation Studies

Ablation results highlighted the critical role of the rotation-invariant loss. Removing this loss from the teacher caused the Mean Per-Joint Position Error (MPJPE) to increase from 85.6mm to 153.5mm, confirming that invariance is the key mechanism for solving projection ambiguity.

Conclusion

ITES demonstrates that decoupling regularization and flexibility through a teacher-student framework is highly effective for unsupervised 3D pose estimation. By combining rotation-invariance for stability and rotation-equivariance for detail, the model overcomes the limitations of single-network architectures. Future work includes incorporating temporal dynamics from video sequences to further resolve depth ambiguities and extending the framework to multi-person scenarios.

Summary updated

Abstract

We propose a novel method based on teacher-student learning framework for 3D human pose estimation without any 3D annotation or side information. To solve this unsupervised-learning problem, the teacher network adopts pose-dictionary-based modeling for regularization to estimate a physically plausible 3D pose. To handle the decomposition ambiguity in the teacher network, we propose a cycle-consistent architecture promoting a 3D rotation-invariant property to train the teacher network. To further improve the estimation accuracy, the student network adopts a novel graph convolution network for flexibility to directly estimate the 3D coordinates. Another cycle-consistent architecture promoting 3D rotation-equivariant property is adopted to exploit geometry consistency, together with knowledge distillation from the teacher network to improve the pose estimation performance. We conduct extensive experiments on Human3.6M and MPI-INF-3DHP. Our method reduces the 3D joint prediction error by 11.4% compared to state-of-the-art unsupervised methods and also outperforms many weakly-supervised methods that use side information on Human3.6M. Code will be available at https://github.com/sjtuxcx/ITES.
BibTeX
@inproceedings{xu2021invariant,
  author = {Chenxin Xu and Siheng Chen and Maosen Li and Ya Zhang},
  title = {{Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16409},
}
Figure