Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks

Huy-Hieu Pham, Louahdi Khoudour, Alain Crouzil, Pablo Zegers, Sergio A. Velastin

TL;DR

This paper introduces SPMF, a novel skeleton-based representation that encodes human actions as color images by combining pose and motion information. This representation is then fed into a deep convolutional neural network (Inception-ResNet) to achieve state-of-the-art 3D action recognition with reduced computational cost.

What Problem Does This Paper Address?

Recognizing human actions from video is a challenging task in computer vision, especially in 3D. While depth cameras provide skeletal data that is robust to lighting and viewpoint changes, effectively representing and learning from this skeletal data remains a hurdle. Previous methods either relied on hand-crafted features, which are data-dependent and require extensive engineering, or used Recurrent Neural Networks (RNNs) on raw skeleton sequences, which can be complex, prone to overfitting, and may not extract high-level features effectively. This paper aims to bridge this gap by proposing a new, robust representation and a powerful deep learning architecture for 3D action recognition.

Methods

The core innovation of this paper is the Skeleton Pose-Motion Feature (SPMF) representation. SPMF is constructed by combining two key aspects of human movement:

  • Pose Features (PFs): These capture the static configuration of a skeleton at a given time. They are derived from joint-joint Euclidean distances and joint-joint orientations. To make this representation more manageable for deep learning, these distances are normalized and then encoded into a greyscale image, with changes in distances represented by a discrete color palette (JET scale).
  • Motion Features (MFs): These capture the dynamics of movement between consecutive frames. Similar to PFs, they are based on joint-joint Euclidean distances and orientations, but calculated between frames t and t+1. These are also encoded into color matrices.

These PFs and MFs are then concatenated chronologically to form the SPMF, which represents an entire skeleton sequence as a single "image."

For learning and classification, the authors propose a deep convolutional neural network (D-CNN) based on the Inception Residual Network (Inception-ResNet) architecture. This choice leverages the strengths of both Inception (efficient multi-scale feature extraction) and Residual Networks (enabling deeper networks by mitigating vanishing gradients and degradation). The Inception-ResNet is trained end-to-end to classify actions from the SPMF images.

Datasets

The proposed method is evaluated on two widely used 3D action recognition datasets:

  • MSR Action3D: This dataset, captured using a Kinect sensor, contains 20 actions performed by 10 subjects. The evaluation uses standard protocols with subsets AS1, AS2, and AS3, where half the data is used for training and the other half for testing.
  • NTU-RGB+D: This is a larger and more challenging dataset, also captured with Kinect sensors. It includes 60 action classes performed by 40 subjects, with over 56,000 videos. Two evaluation criteria are used: Cross-Subject (training on a subset of subjects, testing on the rest) and Cross-View (training on data from cameras 2 and 3, testing on camera 1).

Key Insights and Ideas

  • Transforming Skeletal Data into Images: The key insight is that skeletal movement, with its spatial-temporal dynamics, can be effectively transformed into a 2D image-like representation (SPMF). This allows leveraging the power of Convolutional Neural Networks (CNNs), which excel at image processing.
  • Combining Pose and Motion is Crucial: The paper emphasizes that both the static postures (poses) and their temporal evolution (motions) are essential for robust action recognition. The SPMF representation explicitly captures both.
  • Color Encoding for Dynamics: Using a discrete color palette (JET scale) to represent normalized joint distances and orientations allows the network to distinguish between different movement speeds and patterns, adding a rich layer of information beyond simple greyscale.
  • Inception-ResNet for Deep Feature Learning: The choice of Inception-ResNet is motivated by its ability to learn complex features from deep architectures while maintaining computational efficiency, which is crucial for handling the potentially large SPMF representations.

Key Results

The proposed SPMF representation combined with Inception-ResNet achieves state-of-the-art results on both datasets:

  • MSR Action3D: The SPMF Inception-ResNet-222 configuration achieved an average accuracy of 98.56%, outperforming previous methods.
  • NTU-RGB+D: The method achieved 78.89% accuracy on the Cross-Subject evaluation and 86.15% on the Cross-View evaluation, also surpassing prior state-of-the-art.
  • Computational Efficiency: The paper highlights that the method is not only accurate but also computationally efficient, with a fast training and testing phase, making it suitable for real-time applications.

Limitations and Future Work

  • Limitations: While not explicitly detailed as limitations, the paper implicitly acknowledges the complexity of deep networks and the potential for overfitting, which is why they adopted the Inception-ResNet architecture. The reliance on a specific color mapping (JET scale) might also be a point of exploration for further improvements.
  • Future Work: The authors plan to expand their study by improving the color encoding algorithm. Specifically, they aim to incorporate an image enhancement method to generate even more discriminative features from the image representations, potentially leading to further accuracy gains.

Summary updated

Abstract

We propose a novel skeleton-based representation for 3D action recognition in videos using Deep Convolutional Neural Networks (D-CNNs). Two key issues have been addressed: First, how to construct a robust representation that easily captures the spatial-temporal evolutions of motions from skeleton sequences. Second, how to design D-CNNs capable of learning discriminative features from the new representation in a effective manner. To address these tasks, a skeleton-based representation, namely, SPMF (Skeleton Pose-Motion Feature) is proposed. The SPMFs are built from two of the most important properties of a human action: postures and their motions. Therefore, they are able to effectively represent complex actions. For learning and recognition tasks, we design and optimize new D-CNNs based on the idea of Inception Residual networks to predict actions from SPMFs. Our method is evaluated on two challenging datasets including MSR Action3D and NTU-RGB+D. Experimental results indicated that the proposed method surpasses state-of-the-art methods whilst requiring less computation.
BibTeX
@inproceedings{pham2018skeletal,
  author = {Huy-Hieu Pham and Louahdi Khoudour and Alain Crouzil and Pablo Zegers and Sergio A. Velastin},
  title = {{Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=NAnMyLzLHr},
}
Figure