A union of deep learning and swarm-based optimization for 3D human action recognition

Hritam Basak, Rohit Kundu, Pawan Kumar Singh, Muhammad Fazal Ijaz, Marcin Woźniak, Ram SarkarScientific 2022 · DOI

Abstract

Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models.

Introduction

Human Action Recognition (HAR) is a cornerstone of modern computer vision. While RGB video is a common data source, 3D skeleton data—which tracks joint coordinates—is increasingly preferred. It is computationally lighter and resilient to environmental factors like lighting changes or background clutter. However, effectively processing these "skeletal sticks" remains a challenge. Traditional methods like Recurrent Neural Networks (RNNs) often struggle with complex spatial structures, while newer Graph Convolutional Networks (GCNs) can face scalability issues.

To bridge this gap, researchers have turned to image encoding: transforming joint movements into visual patterns that standard Convolutional Neural Networks (CNNs) can interpret. However, existing encoding methods often produce high-dimensional images with redundant information. This paper introduces DSwarm-Net, a hybrid framework designed to optimize this process by combining deep learning with swarm intelligence.

The DSwarm-Net Framework

The proposed DSwarm-Net architecture follows a multi-stage pipeline designed for efficiency and accuracy. The process begins with raw skeletal data and culminates in a final action prediction using a Support Vector Machine (SVM) classifier.

The DSwarm-Net architecture pipeline

Spatio-Temporal Feature Encoding

Instead of relying on raw coordinates, the framework extracts four distinct feature types to capture the nuances of human movement:

  1. Distance: Spatial relationships between joints.
  2. Distance Velocity: The rate of change in joint distances over time.
  3. Angle: Angular relationships between connected joints.
  4. Angle Velocity: The rate of change in joint angles.

These features are mapped into grayscale images. To maintain computational efficiency, the authors employ depth-wise stacking. Distance and Distance Velocity are combined into a single 2-channel "compact distance input," while Angle and Angle Velocity form a second 2-channel input. This strategy reduces the number of parallel CNN branches required from four to two.

Spatio-temporal feature encoded images: (a) Distance, (b) Distance Velocity, (c) Angle, and (d) Angle Velocity

Deep Feature Extraction and Optimization

The 2-channel images are processed by a modified Inception-ResNet model. Notably, the authors trained this network from scratch. They argue that because encoded skeletal patterns differ fundamentally from natural scenes (like those in ImageNet), standard transfer learning is less effective than learning the unique visual "grammar" of skeletal features.

After extracting 4,096 deep features from the CNN's pre-final layer, the model employs the Ant Lion Optimizer (ALO). This nature-inspired metaheuristic performs feature selection to eliminate non-informative or misleading data. This step is crucial; experimental results showed that accuracy on the UTD-MHAD dataset jumped from approximately 89% to over 98% simply by adding the ALO optimization step.

Experimental Results

DSwarm-Net was evaluated on three benchmark datasets:

  • UTD-MHAD: Achieved 98.13% accuracy, outperforming previous models like SAKDN.
  • HDM05: Reached 90.67%, leading over GCN-based methods such as PB-GCN (88.2%).
  • NTU RGB+D 60: Achieved 85.45% (Cross-Subject) and 89.98% (Cross-View), remaining highly competitive against more complex architectures.

Statistical analysis using McNemar’s test confirmed that the improvements gained through the Ant Lion Optimizer were statistically significant compared to other optimizers like Particle Swarm Optimization (PSO).

Conclusion and Future Work

DSwarm-Net demonstrates that marrying deep learning with swarm-based optimization can significantly enhance 3D action recognition. While the model excels at general action classification, it faces challenges with "closely-knit" classes—actions with nearly identical joint movements, such as "Reading" versus "Writing."

Future research may focus on incorporating 3D flow information or multi-modal data (combining skeleton data with RGB or Infrared) to capture the subtle nuances required to distinguish these highly similar actions.

Summary updated

Abstract

Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models.
BibTeX
@inproceedings{basak2022union,
  author = {Hritam Basak and Rohit Kundu and Pawan Kumar Singh and Muhammad Fazal Ijaz and Marcin Woźniak and Ram Sarkar},
  title = {{A union of deep learning and swarm-based optimization for 3D human action recognition}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-09293-8},
}
Figure