Unsupervised Human 3D Pose Representation with Viewpoint and Pose Disentanglement

Qiang Nie, Ziwei Liu, Yunhui Liu

Introduction

Understanding 3D human pose is a fundamental challenge in computer vision, with applications ranging from healthcare robotics to sports analytics. Two primary hurdles have historically limited performance: viewpoint dependency, where the same pose appears different from varying camera angles, and structural complexity, as human skeletons are kinematic chains rather than simple collections of points.

This research introduces a novel unsupervised method to learn 3D human pose representations by "disentangling" the intrinsic pose (what the person is doing) from the viewpoint (where the camera is looking). By leveraging a specialized tree-structured neural network, the model captures the natural physics of the human skeleton, achieving state-of-the-art results with a remarkably small memory footprint.

The SeBiReNet Architecture

To address the structural complexity of the human body, the authors developed the Sequential Bidirectional Recursive Network (SeBiReNet). Unlike standard neural networks that treat skeleton coordinates as flat lists, SeBiReNet models the skeleton as a tree structure.

The SeBiReNet architecture uses recursive and diffuse paths to propagate information through the skeletal tree.

As shown in the figure above, the network consists of two primary subnets:

  1. Recursive Subnet: Information flows from the extremities (fingers and toes) toward the torso, aggregating local joint data into global features.
  2. Diffuse Subnet: Information flows from the torso back out to the limbs, ensuring that motion and physical constraints propagate naturally through the kinematic chain.

This architecture mimics biological constraints, allowing the model to learn universal human features with only 0.27 million parameters—a fraction of the size of competing models.

Disentangled Representation Learning

The core of the method is a Siamese Denoising Autoencoder designed to separate pose-dependent features from view-dependent features. The model is trained to "denoise" skeletons—taking inputs where joints have been moved to illegal positions and reconstructing the correct, physically plausible pose.

The Siamese Denoising Autoencoder architecture facilitates the disentanglement of pose-dependent ($h_{vi}$) and view-dependent ($h_v$) features.

During training, the model processes the same pose from two different random rotations. The latent representation is explicitly split into two parts:

  • Pose-dependent ($h_{vi}$): Captures the view-invariant intrinsic action.
  • View-dependent ($h_v$): Captures the specific camera angle.

By penalizing the model if the pose feature changes between rotations or if the view feature fails to behave like a proper rotation matrix, the system forces a clean disentanglement. This approach preserves more intrinsic information than traditional "view-invariant" methods that simply discard viewpoint data.

Experimental Results

The model was evaluated on major benchmarks including Cambridge-Imperial APE, Northwestern-UCLA (N-UCLA), and NTU RGB+D. Key findings include:

  • Superior Denoising: SeBiReNet improved the Mean Per Joint Position Error (MPJPE) by 35% compared to standard architectures, demonstrating its superior grasp of skeletal physics.
  • Action Recognition: On the N-UCLA dataset, the model achieved 80.30% accuracy in unsupervised action recognition, significantly outperforming previous state-of-the-art unsupervised methods (62.5%) and even rivaling some supervised approaches.
  • Generalization: Models trained on one dataset successfully denoised skeletons from others, proving that the learned features are universal to human anatomy rather than specific to a single data distribution.

Conclusion and Future Work

By combining structural skeletal priors with explicit feature disentanglement, this method provides a robust, efficient framework for 3D pose representation. The research demonstrates that modeling the human body as a tree is significantly more effective than "black box" approaches.

Future work will focus on extending this framework to 3D pose estimation directly from 2D images and exploring "view transfer"—the ability to realistically re-render human actions from entirely new camera angles within the latent space.

Summary updated

BibTeX
@inproceedings{nie2020unsupervised,
  author = {Qiang Nie and Ziwei Liu and Yunhui Liu},
  title = {{Unsupervised Human 3D Pose Representation with Viewpoint and Pose Disentanglement}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58529-7\_7},
}
Figure