Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold

Jogendra Nath Kundu, Maharshi Gor, Phani Krishna Uppala, R. Venkatesh Babu

TL;DR

This paper introduces a novel unsupervised framework, EnGAN-PoseRNN, that learns to represent human actions as trajectories in a continuous pose embedding manifold. By disentangling pose representation from action modeling and leveraging a GAN-based approach, it achieves state-of-the-art transferability for action recognition tasks, even with limited labeled data.

What Problem Does This Paper Address?

Action recognition from 3D skeletal data has seen great success with supervised deep learning methods. However, these approaches heavily rely on large, annotated datasets, which are expensive and time-consuming to create. The authors argue that unsupervised feature learning can lead to more generalizable representations that are useful across various applications beyond just action recognition, such as motion transfer or biomechanical analysis. Existing end-to-end unsupervised methods often struggle to effectively disentangle the learning of individual poses from the modeling of temporal action dynamics, leading to less scalable and transferable representations. This paper aims to address this by proposing a method that first learns a robust, continuous pose embedding manifold and then models action dynamics as trajectories within this learned space.

Methods

The proposed framework consists of two main components: EnGAN for pose embedding and PoseRNN for trajectory modeling.

EnGAN (Encoder GAN): This is a novel generative adversarial network designed to learn a continuous latent space for 3D human poses. Unlike standard GANs, EnGAN incorporates an encoder that can directly map a pose to its latent representation (one-shot inference). This allows for efficient generation of plausible pose interpolations within the learned manifold. The training involves a cyclic autoencoder to improve reconstruction quality, followed by an adversarial loss to enforce a continuous embedding space. A key aspect is the disentanglement of pose representation learning from action modeling.

PoseRNN: This component models human actions as trajectories in the pose embedding space learned by EnGAN. It utilizes a stacked, bidirectional RNN auto-encoder architecture. The encoder learns a trajectory embedding from a sequence of pose embeddings, and the decoder reconstructs the pose embedding sequence. To improve temporal regularity, a first-order gradient loss is introduced. Additionally, a hierarchical feature fusion technique is employed to combine local joint-level features with global pose embeddings, enabling better modeling of fine-grained actions.

Datasets

The paper uses the following datasets for training and evaluation:

  • NTU RGB+D Dataset: A large-scale dataset for human action recognition with 60 action categories and skeleton data. Used for training and evaluating the proposed framework.
  • SBU Kinect Interaction Dataset: An interaction dataset with 282 sequences across 8 classes. Used to demonstrate the transferability of learned representations.
  • PKU-MMD Dataset: A dataset with 1076 video sequences across 51 action categories, used for training EnGAN.

Key Insights and Ideas

  • Disentangling Pose and Action: Separating the learning of individual pose representations from the modeling of action dynamics as trajectories in a latent space leads to more effective and transferable representations.
  • Continuous Pose Manifold: Using a GAN-based approach (EnGAN) with an encoder allows for learning a continuous, interpolatable pose embedding space, which is crucial for modeling smooth motion trajectories.
  • Hierarchical Feature Fusion: Combining local limb features with global pose embeddings in PoseRNN significantly improves the ability to model fine-grained actions that involve specific joint movements.
  • Unsupervised Transferability: The learned pose embedding and trajectory representations demonstrate strong transferability to new datasets and tasks, even outperforming some supervised methods in semi-supervised settings.

Key Results

  • Reconstruction Quality: EnGAN-PoseRNN achieves competitive reconstruction losses for pose sequences, even for longer sequences (up to 120 frames), as shown in Table 3.
  • Action Recognition Performance: On the NTU dataset, the proposed EnGAN-PoseRNN achieves state-of-the-art results in unsupervised settings (78.7% accuracy on Cross-Subject, 86.5% on Cross-View) and competitive results in semi-supervised settings (77.0% and 88.2% respectively), outperforming other unsupervised methods like VAE-PoseRNN and Holden et al. [8].
  • Transferability: When transferring learned features from NTU to the SBU dataset, EnGAN-PoseRNN achieves the highest accuracy in both unsupervised (77.0%) and semi-supervised (88.2%) transfer settings, surpassing supervised baselines like ST-LSTM and TS-LSTM.
  • Qualitative Results: Visualizations of pose reconstructions and interpolations in the pose embedding space (Figure 4) demonstrate the framework's ability to generate plausible and smooth human poses.

Limitations and Future Work

  • Computational Cost: While the framework disentangles tasks, the overall training process involving GANs and RNNs can still be computationally intensive.
  • Fine-grained Action Complexity: While the hierarchical fusion helps, modeling extremely complex or subtle fine-grained actions might still pose a challenge.
  • Future Work: The authors suggest exploring applications in 3D pose estimation and motion transfer to humanoid robots. Further investigation into more efficient architectures and potentially incorporating attention mechanisms could also be beneficial.

Summary updated

Abstract

An unsupervised human action modeling framework can provide useful pose-sequence representation, which can be utilized in a variety of pose analysis applications. In this work we propose a novel temporal pose-sequence modeling framework, which can embed the dynamics of 3D human-skeleton joints to a continuous latent space in an efficient manner. In contrast to end-to-end framework explored by previous works, we disentangle the task of individual pose representation learning from the task of learning actions as a trajectory in pose embedding space. In order to realize a continuous pose embedding manifold with improved reconstructions, we propose an unsupervised, manifold learning procedure named Encoder GAN, (or EnGAN). Further, we use the pose embeddings generated by EnGAN to model human actions using a bidirectional RNN auto-encoder architecture, PoseRNN. We introduce first-order gradient loss to explicitly enforce temporal regularity in the predicted motion sequence. A hierarchical feature fusion technique is also investigated for simultaneous modeling of local skeleton joints along with global pose variations. We demonstrate state-of-the-art transfer-ability of the learned representation against other supervisedly and unsupervisedly learned motion embeddings for the task of fine-grained action recognition on SBU interaction dataset. Further, we show the qualitative strengths of the proposed framework by visualizing skeleton pose reconstructions and interpolations in pose-embedding space, and low dimensional principal component projections of the reconstructed pose trajectories.
BibTeX
@inproceedings{kundu2018unsupervised,
  author = {Jogendra Nath Kundu and Maharshi Gor and Phani Krishna Uppala and R. Venkatesh Babu},
  title = {{Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=5C69ztzwfO},
}
Figure