Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning

Mengyuan Liu, Fanyang Meng, Yongsheng LiangCyborg 2022 · DOI

TL;DR

This paper introduces a novel "Pose Decoupled Flow" (PDF) network for unsupervised 3D human action recognition. By decoupling motion into direction and magnitude, and leveraging this "pose flow" as a supervision signal, their method learns more discriminative representations than traditional autoencoders that only reconstruct pose sequences. Their generalized PDF network (PDF-G) further improves performance by incorporating shape information, achieving state-of-the-art results on challenging datasets.

What Problem Does This Paper Address?

Recognizing human actions from 3D skeleton data is a crucial task in computer vision with applications in surveillance, healthcare, and human-robot interaction. While supervised methods have achieved high accuracy, they require extensive manual labeling of pose sequences, which is time-consuming and costly. Unsupervised methods aim to learn action representations without these labels. Traditional unsupervised approaches often use autoencoders to reconstruct the input pose sequence. However, these methods tend to focus on human shape and struggle to capture subtle motion cues that are vital for distinguishing similar actions (e.g., "sitting down" vs. "standing up" rely on motion direction, while "running" vs. "jogging" rely on motion norm). This paper addresses the limitation of existing unsupervised methods by proposing a novel approach that explicitly models and learns from motion information, specifically by decoupling the components of motion.

Methods

The core of the proposed method is the Pose Decoupled Flow (PDF) network. Instead of directly reconstructing the input pose sequence, the authors introduce pose flow, a handcrafted motion feature inspired by optical flow. Pose flow captures the movement between poses. To improve discriminative power, they further decouple pose flow into two components: pose magnitude flow (how much joints move) and pose orientation flow (in which direction joints move).

The authors propose three main network architectures:

  • Explicit PDF network (PDF-E): This network uses a shared encoder to generate a latent representation and two separate decoders to reconstruct the pose magnitude flow and pose orientation flow independently.
  • Implicit PDF network (PDF-I): This version also uses two decoders but generates intermediate variables that are then used to infer the pose magnitude and orientation flow. It introduces an additional constraint loss to regularize these intermediate variables.
  • Generalized PDF network (PDF-G): This is the most advanced version. It uses pose flow (magnitude and orientation) as the primary supervision signal but also incorporates the original pose sequence as an additional constraint. This allows the network to learn both motion and shape information simultaneously, leading to more robust representations.

The supervision signal generation involves calculating pose flow from the input pose sequences. For evaluation, the learned latent representations are fed into a simple k-Nearest Neighbors (KNN) classifier.

Datasets

The paper evaluates their methods on two large-scale and challenging 3D action recognition datasets:

  • NTU RGB+D 60 (NTU-60): Contains 60 action classes, 40 subjects, and 56,880 pose sequences. Evaluated using Cross Subject (CSub) and Cross View (CView) protocols.
  • NTU RGB+D 120 (NTU-120): Contains 120 action classes, 106 subjects, and 114,480 pose sequences. Evaluated using Cross Subject (CSub) and Cross Setup (CSet) protocols.

No new datasets were introduced.

Key Insights and Ideas

  • Motion is Key, and Decoupling it is Better: The paper strongly emphasizes that motion information is crucial for action recognition, and simply reconstructing pose sequences is insufficient. Decoupling motion into direction and magnitude (pose flow) allows the network to learn more discriminative features, especially for actions that differ subtly in these aspects.
  • Pose Flow as a Powerful Supervision Signal: Using pose flow (especially reference pose flow, RPF) as a supervision signal significantly outperforms using raw pose sequences for unsupervised action recognition. This highlights the effectiveness of motion-based supervision.
  • Combining Motion and Shape is Optimal: The generalized PDF network (PDF-G) demonstrates that integrating both motion (pose flow) and shape (original pose sequence) information leads to the best performance, achieving state-of-the-art results. This suggests that a holistic representation is most effective.
  • Unsupervised Learning Can Rival Supervised Methods: The paper shows that their unsupervised approach, particularly PDF-G, can achieve performance comparable to, and in some cases even surpass, supervised methods on these challenging datasets.

Key Results

  • The proposed PDF-E network outperforms baseline methods using pose sequence reconstruction by up to 5.7% on NTU-60 (CSub).
  • The generalized PDF-G network achieves state-of-the-art performance among unsupervised methods, outperforming the baseline by 6.1% on NTU-60 (CSub).
  • On NTU-60 (CSub), PDF-G achieves 59.7% accuracy, and PDF-G* (with an enhanced encoder) reaches 60.4%.
  • PDF-G outperforms previous unsupervised pose-based methods by at least 7% on NTU-60 (CSub).
  • Remarkably, the proposed unsupervised methods (PDF-G and PDF-G) achieve accuracies comparable to or better than many supervised pose-based methods, such as P-LSTM and Soft RNN. For instance, PDF-G achieves 51.3% on NTU-120 (CSet), while P-LSTM achieves 26.3%.

Limitations and Future Work

  • Noise Sensitivity: The authors acknowledge that improving the robustness of their method to noises in pose sequences is a direction for future work.
  • Pose Estimation Dependency: The performance of their method is inherently dependent on the quality of the input 3D pose estimation.
  • Future Directions: The authors suggest extending their work to related fields like pose-based human action retrieval and few-shot learning for action recognition. They also plan to explore more complex encoders to further boost performance.

Summary updated

Abstract

Human action representation is derived from the description of human shape and motion. The traditional unsupervised 3-dimensional (3D) human action representation learning method uses a recurrent neural network (RNN)-based autoencoder to reconstruct the input pose sequence and then takes the midlevel feature of the autoencoder as representation. Although RNN can implicitly learn a certain amount of motion information, the extracted representation mainly describes the human shape and is insufficient to describe motion information. Therefore, we first present a handcrafted motion feature called pose flow to guide the reconstruction of the autoencoder, whose midlevel feature is expected to describe motion information. The performance is limited as we observe that actions can be distinctive in either motion direction or motion norm. For example, we can distinguish "sitting down" and "standing up" from motion direction yet distinguish "running" and "jogging" from motion norm. In these cases, it is difficult to learn distinctive features from pose flow where direction and norm are mixed. To this end, we present an explicit pose decoupled flow network (PDF-E) to learn from direction and norm in a multi-task learning framework, where 1 encoder is used to generate representation and 2 decoders are used to generating direction and norm, respectively. Further, we use reconstructing the input pose sequence as an additional constraint and present a generalized PDF network (PDF-G) to learn both motion and shape information, which achieves state-of-the-art performances on large-scale and challenging 3D action recognition datasets including the NTU RGB+D 60 dataset and NTU RGB+D 120 dataset.
BibTeX
@inproceedings{liu2022generalized,
  author = {Mengyuan Liu and Fanyang Meng and Yongsheng Liang},
  title = {{Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning}},
  booktitle = {Cyborg},
  year = {2022},
  doi = {10.34133/cbsystems.0002},
}
Figure