Investigation of different skeleton features for CNN-based 3D action recognition

Zewei Ding, Pichao Wang, Philip O. Ogunbona, Wanqing Li

Abstract

Human action recognition using 3D skeleton data offers robustness against environmental variations but presents challenges in effectively modeling spatial-temporal dynamics. This paper investigates a framework for transforming skeleton sequences into 2D "texture images" to leverage the representational power of Convolutional Neural Networks (CNNs). By systematically evaluating five types of geometric features—including joint-joint distances, vectors, and orientations—and various encoding strategies, the proposed method achieves a state-of-the-art accuracy of 82.31% on the NTU RGB+D dataset. The results highlight the importance of multi-feature fusion in capturing complementary aspects of human motion.

Introduction

Action recognition is a fundamental task in computer vision with applications ranging from surveillance to human-computer interaction. While 3D skeleton data is inherently robust to lighting and viewpoint changes, traditional sequence modeling often relies on Recurrent Neural Networks (RNNs). However, RNNs can suffer from overfitting and may not fully capture complex spatial relationships. CNNs offer a powerful alternative but require 2D input. This research addresses the challenge of "flattening" temporal 3D joint coordinates into informative 2D images without losing critical motion information.

Proposed Methodology

The authors propose a multi-stream pipeline that converts skeleton sequences into color images for processing by a standard CNN architecture. The process involves three key stages: feature extraction, selection, and encoding.

Overview of the proposed action recognition pipeline, from skeleton sequences to fused CNN predictions.

Feature Extraction and Selection

Rather than using raw coordinates, the framework calculates five geometric relationships:

  1. Joint-Joint Distances (JJd) and Vectors (JJv)
  2. Joint-Joint Orientations (JJo)
  3. Joint-Line Distances (JLd)
  4. Line-Line Angles (LLa)

To manage the high dimensionality of these features, the authors employed selection strategies (JS1-JS3) to focus on the most informative joints, such as those at the extremities of limbs which exhibit the most significant movement.

Feature Encoding and Fusion

The selected features are mapped into RGB channels to create "texture images." One innovative encoding method (EM3) uses different color channels to represent interactions between primary and auxiliary subjects (e.g., two people hugging). The final system trains 13 separate CNNs—one for each feature/encoding pair—and fuses their outputs using element-wise multiplication to make a final prediction.

Key Insights: Motion as Visual Texture

A core contribution of this work is the conceptualization of human movement as a visual texture. By arranging joint indices along one axis and time along the other, specific actions generate distinct, recognizable patterns that a CNN can identify regardless of temporal shifts.

Visual representation of different skeleton features encoded as 2D texture images for various action classes.

The study found that relative spatial relationships, particularly Joint-Joint Vectors (JJv), were more discriminative than absolute coordinates. Furthermore, the significant jump in accuracy from individual features (ranging from 55% to 69%) to the fused model (82.31%) underscores that different geometric perspectives provide complementary information essential for accurate recognition.

Experimental Results

The method was rigorously tested on the NTU RGB+D dataset using the cross-view protocol. Achieving 82.31% accuracy, the proposed approach significantly outperformed established baselines, including Deep RNNs (64.09%) and previous CNN-based methods like Joint Trajectory Maps (75.20%). These results demonstrate that well-engineered geometric textures can effectively bridge the gap between 3D skeleton sequences and 2D deep learning architectures.

Conclusion

This investigation proves that transforming 3D skeleton data into 2D geometric textures is a highly effective strategy for action recognition. While the current multi-CNN pipeline is computationally intensive, the success of the fusion approach suggests that future work should focus on more efficient end-to-end architectures that can automatically learn these optimal spatial-temporal representations from raw data.

Summary updated

BibTeX
@inproceedings{ding2017investigation,
  author = {Zewei Ding and Pichao Wang and Philip O. Ogunbona and Wanqing Li},
  title = {{Investigation of different skeleton features for CNN-based 3D action recognition}},
  booktitle = {Workshops @ ICME},
  year = {2017},
  doi = {10.1109/icmew.2017.8026286},
}
Figure