Abstract
Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models.
Introduction
Human Action Recognition (HAR) is a cornerstone of modern computer vision. While RGB video is a common data source, 3D skeleton data—which tracks joint coordinates—is increasingly preferred. It is computationally lighter and resilient to environmental factors like lighting changes or background clutter. However, effectively processing these "skeletal sticks" remains a challenge. Traditional methods like Recurrent Neural Networks (RNNs) often struggle with complex spatial structures, while newer Graph Convolutional Networks (GCNs) can face scalability issues.
To bridge this gap, researchers have turned to image encoding: transforming joint movements into visual patterns that standard Convolutional Neural Networks (CNNs) can interpret. However, existing encoding methods often produce high-dimensional images with redundant information. This paper introduces DSwarm-Net, a hybrid framework designed to optimize this process by combining deep learning with swarm intelligence.
The DSwarm-Net Framework
The proposed DSwarm-Net architecture follows a multi-stage pipeline designed for efficiency and accuracy. The process begins with raw skeletal data and culminates in a final action prediction using a Support Vector Machine (SVM) classifier.
Spatio-Temporal Feature Encoding
Instead of relying on raw coordinates, the framework extracts four distinct feature types to capture the nuances of human movement:
- Distance: Spatial relationships between joints.
- Distance Velocity: The rate of change in joint distances over time.
- Angle: Angular relationships between connected joints.
- Angle Velocity: The rate of change in joint angles.
These features are mapped into grayscale images. To maintain computational efficiency, the authors employ depth-wise stacking. Distance and Distance Velocity are combined into a single 2-channel "compact distance input," while Angle and Angle Velocity form a second 2-channel input. This strategy reduces the number of parallel CNN branches required from four to two.
Deep Feature Extraction and Optimization
The 2-channel images are processed by a modified Inception-ResNet model. Notably, the authors trained this network from scratch. They argue that because encoded skeletal patterns differ fundamentally from natural scenes (like those in ImageNet), standard transfer learning is less effective than learning the unique visual "grammar" of skeletal features.
After extracting 4,096 deep features from the CNN's pre-final layer, the model employs the Ant Lion Optimizer (ALO). This nature-inspired metaheuristic performs feature selection to eliminate non-informative or misleading data. This step is crucial; experimental results showed that accuracy on the UTD-MHAD dataset jumped from approximately 89% to over 98% simply by adding the ALO optimization step.
Experimental Results
DSwarm-Net was evaluated on three benchmark datasets:
- UTD-MHAD: Achieved 98.13% accuracy, outperforming previous models like SAKDN.
- HDM05: Reached 90.67%, leading over GCN-based methods such as PB-GCN (88.2%).
- NTU RGB+D 60: Achieved 85.45% (Cross-Subject) and 89.98% (Cross-View), remaining highly competitive against more complex architectures.
Statistical analysis using McNemar’s test confirmed that the improvements gained through the Ant Lion Optimizer were statistically significant compared to other optimizers like Particle Swarm Optimization (PSO).
Conclusion and Future Work
DSwarm-Net demonstrates that marrying deep learning with swarm-based optimization can significantly enhance 3D action recognition. While the model excels at general action classification, it faces challenges with "closely-knit" classes—actions with nearly identical joint movements, such as "Reading" versus "Writing."
Future research may focus on incorporating 3D flow information or multi-modal data (combining skeleton data with RGB or Infrared) to capture the subtle nuances required to distinguish these highly similar actions.
Summary updated