TL;DR
This paper introduces a novel unsupervised framework, EnGAN-PoseRNN, that learns to represent human actions as trajectories in a continuous pose embedding manifold. By disentangling pose representation from action modeling and leveraging a GAN-based approach, it achieves state-of-the-art transferability for action recognition tasks, even with limited labeled data.
What Problem Does This Paper Address?
Action recognition from 3D skeletal data has seen great success with supervised deep learning methods. However, these approaches heavily rely on large, annotated datasets, which are expensive and time-consuming to create. The authors argue that unsupervised feature learning can lead to more generalizable representations that are useful across various applications beyond just action recognition, such as motion transfer or biomechanical analysis. Existing end-to-end unsupervised methods often struggle to effectively disentangle the learning of individual poses from the modeling of temporal action dynamics, leading to less scalable and transferable representations. This paper aims to address this by proposing a method that first learns a robust, continuous pose embedding manifold and then models action dynamics as trajectories within this learned space.
Methods
The proposed framework consists of two main components: EnGAN for pose embedding and PoseRNN for trajectory modeling.
EnGAN (Encoder GAN): This is a novel generative adversarial network designed to learn a continuous latent space for 3D human poses. Unlike standard GANs, EnGAN incorporates an encoder that can directly map a pose to its latent representation (one-shot inference). This allows for efficient generation of plausible pose interpolations within the learned manifold. The training involves a cyclic autoencoder to improve reconstruction quality, followed by an adversarial loss to enforce a continuous embedding space. A key aspect is the disentanglement of pose representation learning from action modeling.
PoseRNN: This component models human actions as trajectories in the pose embedding space learned by EnGAN. It utilizes a stacked, bidirectional RNN auto-encoder architecture. The encoder learns a trajectory embedding from a sequence of pose embeddings, and the decoder reconstructs the pose embedding sequence. To improve temporal regularity, a first-order gradient loss is introduced. Additionally, a hierarchical feature fusion technique is employed to combine local joint-level features with global pose embeddings, enabling better modeling of fine-grained actions.
Datasets
The paper uses the following datasets for training and evaluation:
- NTU RGB+D Dataset: A large-scale dataset for human action recognition with 60 action categories and skeleton data. Used for training and evaluating the proposed framework.
- SBU Kinect Interaction Dataset: An interaction dataset with 282 sequences across 8 classes. Used to demonstrate the transferability of learned representations.
- PKU-MMD Dataset: A dataset with 1076 video sequences across 51 action categories, used for training EnGAN.
Key Insights and Ideas
- Disentangling Pose and Action: Separating the learning of individual pose representations from the modeling of action dynamics as trajectories in a latent space leads to more effective and transferable representations.
- Continuous Pose Manifold: Using a GAN-based approach (EnGAN) with an encoder allows for learning a continuous, interpolatable pose embedding space, which is crucial for modeling smooth motion trajectories.
- Hierarchical Feature Fusion: Combining local limb features with global pose embeddings in PoseRNN significantly improves the ability to model fine-grained actions that involve specific joint movements.
- Unsupervised Transferability: The learned pose embedding and trajectory representations demonstrate strong transferability to new datasets and tasks, even outperforming some supervised methods in semi-supervised settings.
Key Results
- Reconstruction Quality: EnGAN-PoseRNN achieves competitive reconstruction losses for pose sequences, even for longer sequences (up to 120 frames), as shown in Table 3.
- Action Recognition Performance: On the NTU dataset, the proposed EnGAN-PoseRNN achieves state-of-the-art results in unsupervised settings (78.7% accuracy on Cross-Subject, 86.5% on Cross-View) and competitive results in semi-supervised settings (77.0% and 88.2% respectively), outperforming other unsupervised methods like VAE-PoseRNN and Holden et al. [8].
- Transferability: When transferring learned features from NTU to the SBU dataset, EnGAN-PoseRNN achieves the highest accuracy in both unsupervised (77.0%) and semi-supervised (88.2%) transfer settings, surpassing supervised baselines like ST-LSTM and TS-LSTM.
- Qualitative Results: Visualizations of pose reconstructions and interpolations in the pose embedding space (Figure 4) demonstrate the framework's ability to generate plausible and smooth human poses.
Limitations and Future Work
- Computational Cost: While the framework disentangles tasks, the overall training process involving GANs and RNNs can still be computationally intensive.
- Fine-grained Action Complexity: While the hierarchical fusion helps, modeling extremely complex or subtle fine-grained actions might still pose a challenge.
- Future Work: The authors suggest exploring applications in 3D pose estimation and motion transfer to humanoid robots. Further investigation into more efficient architectures and potentially incorporating attention mechanisms could also be beneficial.
Summary updated