Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement

Qiang Nie, Ziwei Liu, Yunhui Liu

Introduction

Understanding 3D human pose is a cornerstone of computer vision with applications ranging from robotics to surveillance. However, two primary challenges persist: significant viewpoint variation and the complex intrinsic dependencies of the human skeleton. Standard approaches often require extensive labeled data or attempt to discard viewpoint information entirely. This paper proposes a novel unsupervised framework that explicitly disentangles "what the pose is" (view-invariant) from "where it is viewed from" (view-variant).

Methodology

SeBiReNet: A Bio-inspired Architecture

A key contribution of this work is the Sequential Bidirectional Recursive Network (SeBiReNet). Unlike standard Multi-Layer Perceptrons (MLPs) that treat joints as independent points, SeBiReNet leverages the tree-like structure of the human skeleton.

The SeBiReNet architecture, featuring recursive and diffuse subnets that mirror the human skeletal structure.

As shown in the figure above, the network consists of two main components:

  1. Recursive Subnet: Aggregates information from the "fingertips" toward the "root" (pelvis), capturing local-to-global kinematic constraints.
  2. Diffuse Subnet: Distributes information from the root back to the limbs, ensuring global consistency across the skeleton. This bidirectional flow allows the model to learn complex joint dependencies with a fraction of the parameters required by traditional architectures.

Siamese Denoising Autoencoder and Disentanglement

The overall framework utilizes a Siamese Denoising Autoencoder (DAE) to learn robust representations without manual labels. The model is trained to reconstruct clean 3D poses from corrupted inputs, forcing it to internalize biological constraints like bone length and symmetry.

The proposed Siamese Denoising Autoencoder framework for unsupervised pose and viewpoint disentanglement.

The encoder maps the input into a latent space split into two distinct features: $h_{vi}$ (view-invariant pose) and $h_v$ (view-variant viewpoint). To enforce this separation, the authors employ a "view-transfer" strategy: swapping viewpoint features between different views of the same pose and requiring the decoder to reconstruct the original configurations. An orthogonal constraint is also applied to $h_v$ to prevent pose information from leaking into the viewpoint representation.

Experiments and Key Results

The framework was evaluated across several major datasets, including Cambridge-Imperial APE, N-UCLA, and NTU RGB+D.

  • Unsupervised Action Recognition: On the N-UCLA dataset, the model achieved a state-of-the-art unsupervised accuracy of 80.30%, significantly outperforming previous methods and approaching supervised performance levels.
  • Parameter Efficiency: The model is remarkably lightweight. On the NTU RGB+D dataset, it achieved 79.71% accuracy with only 0.27 million parameters, whereas competing state-of-the-art models often require tens of millions.
  • Pose Denoising: SeBiReNet demonstrated superior performance in pose reconstruction, reducing the Mean Per Joint Position Error (MPJPE) to 33.39mm, a 50% improvement over standard MLPs.
  • Transferability: Representations learned on the APE dataset transferred effectively to N-UCLA and NTU without fine-tuning, indicating the model learns universal human kinematic properties.

Conclusion

By combining a bio-inspired recursive network with a disentangled latent space, this framework provides a highly efficient and effective solution for unsupervised 3D pose representation. Future work may involve integrating temporal dynamics directly into the disentanglement process to further enhance action recognition capabilities.

Summary updated

Abstract

Learning a good 3D human pose representation is important for human pose related tasks, e.g. human 3D pose estimation and action recognition. Within all these problems, preserving the intrinsic pose information and adapting to view variations are two critical issues. In this work, we propose a novel Siamese denoising autoencoder to learn a 3D pose representation by disentangling the pose-dependent and view-dependent feature from the human skeleton data, in a fully unsupervised manner. These two disentangled features are utilized together as the representation of the 3D pose. To consider both the kinematic and geometric dependencies, a sequential bidirectional recursive network (SeBiReNet) is further proposed to model the human skeleton data. Extensive experiments demonstrate that the learned representation 1) preserves the intrinsic information of human pose, 2) shows good transferability across datasets and tasks. Notably, our approach achieves state-of-the-art performance on two inherently different tasks: pose denoising and unsupervised action recognition. Code and models are available at: https://github.com/NIEQiang001/unsupervised-human-pose.git .
BibTeX
@inproceedings{nie2020unsupervised,
  author = {Qiang Nie and Ziwei Liu and Yunhui Liu},
  title = {{Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58529-7\_7},
}
Figure