Abstract
This paper introduces Tree Structure Reference Joints Image (TSRJI), a novel 2D image representation for 3D skeleton-based action recognition. By employing a depth-first tree traversal to preserve spatial dependencies and utilizing relative coordinates from stable reference joints, the proposed method transforms temporal skeleton sequences into informative image-like structures. These representations are then processed by a lightweight Convolutional Neural Network (CNN). Experimental results on the NTU RGB+D 60 and 120 datasets demonstrate that TSRJI achieves competitive performance with significantly lower computational overhead compared to deep recurrent or complex convolutional architectures.
Introduction
3D action recognition typically relies on skeleton data—the $(x, y, z)$ coordinates of human joints over time. Traditional approaches generally fall into two categories: Recurrent Neural Networks (RNNs), which excel at modeling temporal sequences but struggle with complex spatial relationships, and Convolutional Neural Networks (CNNs), which are powerful spatial learners but require skeleton data to be flattened into a 2D array. A major challenge in the CNN approach is that linear joint indexing often destroys the semantic relationships between connected body parts (e.g., placing a hand joint far from an elbow joint in the data structure), making it difficult for the network to learn coordinated movements.
Methodology: The TSRJI Representation
To address the limitations of standard skeleton flattening, the authors propose the TSRJI representation, which leverages two core techniques to create more semantically meaningful "skeleton images."
Tree Structure Traversal
Instead of a simple linear list, the method uses a depth-first search (DFS) algorithm to traverse the human skeleton tree. This ensures that joints physically connected in the body—such as the shoulder, elbow, and wrist—remain adjacent in the resulting image rows. By minimizing the structural distance between related joints, the CNN can more effectively extract local spatial features.
Reference Joints Technique
Absolute coordinates are often susceptible to noise and subject positioning. To capture stable motion patterns, the authors calculate the position of every joint relative to four stable "reference joints": the left shoulder, right shoulder, left hip, and right hip. This process generates four distinct coordinate chains for every frame, emphasizing how the extremities move relative to the body's core.
Model Architecture
The TSRJI pipeline is designed for efficiency. Once the four DFS-based chains are generated for each frame, they are stacked over time to create four image matrices ($J \times T \times 3$). These images are fed into a lightweight CNN architecture consisting of only three convolutional layers followed by max-pooling and ReLU activations.
To finalize the classification, the model employs Late Fusion, combining the prediction scores from the four different reference-joint models to produce a final action class label. This multi-view approach allows the model to capture complementary information from different anatomical perspectives.
Key Insights and Results
The model was evaluated on the NTU RGB+D 60 and NTU RGB+D 120 benchmarks, which are among the largest and most challenging datasets in the field.
- Spatial Preservation: Ablation studies showed that the tree-based traversal significantly outperformed random joint ordering, confirming that the topology of the image representation is crucial for CNN performance.
- Efficiency: The compact nature of TSRJI allows for a shallow CNN, making the system faster to train and deploy than deep ResNets or complex LSTMs.
- Performance: On NTU RGB+D 60, the method achieved 73.3% (Cross-Subject) and 80.3% (Cross-View) accuracy. On the NTU 120 dataset, it reached 67.9% accuracy (Cross-Subject) when combined with additional motion features.
- Specific Strengths: The representation proved particularly effective at distinguishing actions involving upper-body movements, as the shoulder reference joints provided a stable anchor for fine-grained arm and hand motions.
Conclusion and Future Work
TSRJI provides a robust and efficient framework for 3D action recognition by bridging the gap between skeleton topology and convolutional processing. While the model currently faces challenges with fine-grained action confusion (e.g., "clapping" vs. "rubbing hands") and object-dependent actions, future work may involve integrating attention mechanisms or extending the representation to 2D skeletons estimated from standard RGB video.
Summary updated