Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints

Carlos Caetano, François Brémond, William Robson Schwartz

Abstract

This paper introduces Tree Structure Reference Joints Image (TSRJI), a novel 2D image representation for 3D skeleton-based action recognition. By employing a depth-first tree traversal to preserve spatial dependencies and utilizing relative coordinates from stable reference joints, the proposed method transforms temporal skeleton sequences into informative image-like structures. These representations are then processed by a lightweight Convolutional Neural Network (CNN). Experimental results on the NTU RGB+D 60 and 120 datasets demonstrate that TSRJI achieves competitive performance with significantly lower computational overhead compared to deep recurrent or complex convolutional architectures.

Introduction

3D action recognition typically relies on skeleton data—the $(x, y, z)$ coordinates of human joints over time. Traditional approaches generally fall into two categories: Recurrent Neural Networks (RNNs), which excel at modeling temporal sequences but struggle with complex spatial relationships, and Convolutional Neural Networks (CNNs), which are powerful spatial learners but require skeleton data to be flattened into a 2D array. A major challenge in the CNN approach is that linear joint indexing often destroys the semantic relationships between connected body parts (e.g., placing a hand joint far from an elbow joint in the data structure), making it difficult for the network to learn coordinated movements.

Methodology: The TSRJI Representation

To address the limitations of standard skeleton flattening, the authors propose the TSRJI representation, which leverages two core techniques to create more semantically meaningful "skeleton images."

Tree Structure Traversal

Instead of a simple linear list, the method uses a depth-first search (DFS) algorithm to traverse the human skeleton tree. This ensures that joints physically connected in the body—such as the shoulder, elbow, and wrist—remain adjacent in the resulting image rows. By minimizing the structural distance between related joints, the CNN can more effectively extract local spatial features.

Reference Joints Technique

Absolute coordinates are often susceptible to noise and subject positioning. To capture stable motion patterns, the authors calculate the position of every joint relative to four stable "reference joints": the left shoulder, right shoulder, left hip, and right hip. This process generates four distinct coordinate chains for every frame, emphasizing how the extremities move relative to the body's core.

The proposed pipeline for generating TSRJI matrices, showing the transition from skeleton tree to depth-first traversal and reference joint computation.

Model Architecture

The TSRJI pipeline is designed for efficiency. Once the four DFS-based chains are generated for each frame, they are stacked over time to create four image matrices ($J \times T \times 3$). These images are fed into a lightweight CNN architecture consisting of only three convolutional layers followed by max-pooling and ReLU activations.

The end-to-end classification pipeline, illustrating the input skeleton sequence, the generated image representation, and the lightweight CNN architecture.

To finalize the classification, the model employs Late Fusion, combining the prediction scores from the four different reference-joint models to produce a final action class label. This multi-view approach allows the model to capture complementary information from different anatomical perspectives.

Key Insights and Results

The model was evaluated on the NTU RGB+D 60 and NTU RGB+D 120 benchmarks, which are among the largest and most challenging datasets in the field.

  • Spatial Preservation: Ablation studies showed that the tree-based traversal significantly outperformed random joint ordering, confirming that the topology of the image representation is crucial for CNN performance.
  • Efficiency: The compact nature of TSRJI allows for a shallow CNN, making the system faster to train and deploy than deep ResNets or complex LSTMs.
  • Performance: On NTU RGB+D 60, the method achieved 73.3% (Cross-Subject) and 80.3% (Cross-View) accuracy. On the NTU 120 dataset, it reached 67.9% accuracy (Cross-Subject) when combined with additional motion features.
  • Specific Strengths: The representation proved particularly effective at distinguishing actions involving upper-body movements, as the shoulder reference joints provided a stable anchor for fine-grained arm and hand motions.

Conclusion and Future Work

TSRJI provides a robust and efficient framework for 3D action recognition by bridging the gap between skeleton topology and convolutional processing. While the model currently faces challenges with fine-grained action confusion (e.g., "clapping" vs. "rubbing hands") and object-dependent actions, future work may involve integrating attention mechanisms or extending the representation to 2D skeletons estimated from standard RGB video.

Summary updated

Abstract

In the last years, the computer vision research community has studied on how to model temporal dynamics in videos to employ 3D human action recognition. To that end, two main baseline approaches have been researched: (i) Recurrent Neural Networks (RNNs) with Long-Short Term Memory (LSTM); and (ii) skeleton image representations used as input to a Convolutional Neural Network (CNN). Although RNN approaches present excellent results, such methods lack the ability to efficiently learn the spatial relations between the skeleton joints. On the other hand, the representations used to feed CNN approaches present the advantage of having the natural ability of learning structural information from 2D arrays (i.e., they learn spatial relations from the skeleton joints). To further improve such representations, we introduce the Tree Structure Reference Joints Image (TSRJI), a novel skeleton image representation to be used as input to CNNs. The proposed representation has the advantage of combining the use of reference joints and a tree structure skeleton. While the former incorporates different spatial relationships between the joints, the latter preserves important spatial relations by traversing a skeleton tree with a depth-first order algorithm. Experimental results demonstrate the effectiveness of the proposed representation for 3D action recognition on two datasets achieving state-of-the-art results on the recent NTU RGB+D 120 dataset.
BibTeX
@inproceedings{caetano2019skeleton,
  author = {Carlos Caetano and François Brémond and William Robson Schwartz},
  title = {{Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints}},
  booktitle = {SIBGRAPI},
  year = {2019},
  url = {https://openreview.net/forum?id=yrQgv097El},
}
Figure