A Novel Skeleton-Based Human Activity Discovery Using Particle Swarm Optimization With Gaussian Mutation

Parham Hadikhani, Daphne Teck Ching Lai, Wee-Hong Ong

TL;DR

This paper introduces a novel, fully unsupervised method for discovering human activities from 3D skeleton data. By combining advanced feature extraction, dimensionality reduction with PCA, and a hybrid Particle Swarm Optimization (PSO) with Gaussian Mutation and K-means (HPGMK) clustering algorithm, the method effectively groups similar activities without requiring any prior labels, achieving superior performance over existing state-of-the-art approaches.

What Problem Does This Paper Address?

Human Activity Recognition (HAR) is a crucial field with applications ranging from human-computer interaction to intelligent transportation. However, most existing HAR systems are supervised, meaning they require large amounts of labeled data to train. Labeling human activity data is a significant bottleneck due to the sheer volume and diversity of human actions. This paper tackles the challenge of human activity discovery, which aims to group similar activities without any prior knowledge of what constitutes each activity. The core problem is to automatically identify and cluster distinct human movements from unlabeled 3D skeleton sequences, overcoming the limitations of supervised methods and the difficulties of manual annotation.

Methods

The proposed framework consists of two main stages: pre-processing and discovery.

Pre-processing: * Keyframe Selection: Instead of using all frames, important frames are selected based on their kinetic energy. This involves identifying frames with local maxima and minima in kinetic energy, which are considered to represent crucial postures and movements, thus reducing redundancy and complexity. * Feature Extraction: A comprehensive set of features is extracted from informative joints (e.g., hands, feet, hips, shoulders, elbows, knees) to capture various aspects of human movement. These include: * Spatial and Temporal Displacement: Measures the movement of joints within a frame and across consecutive frames, as well as relative to a neutral pose. * Statistical Features: Mean and standard deviation of time-domain features to capture how activity evolves over time, particularly distinguishing arm and leg movements. * Orientation Features: Angles and rotation matrices between bones to describe the relative orientation of body parts. * Dimension Reduction: Principal Component Analysis (PCA) is applied to reduce the dimensionality of the extracted features, making the subsequent clustering more efficient and effective by removing redundant information. * Sampling: A sliding time window with overlapping segments is used to create activity samples from the processed skeleton sequences.

Discovery: * Hybrid PSO with Gaussian Mutation and K-means (HPGMK): This is the core clustering algorithm. * Particle Swarm Optimization (PSO): Used for its global search capability to find optimal cluster centers. * Gaussian Mutation: Introduced to enhance the diversity of solutions explored by PSO, preventing premature convergence and local optima. This mutation is applied to the global best particle. * K-means Clustering: Integrated to refine the cluster centroids obtained by PSO. K-means excels at local search and helps overcome PSO's slow convergence when approaching the final solution, leading to better exploitation.

Datasets

The proposed method was evaluated on five publicly available 3D skeleton-based human activity datasets:

  • Cornell Activity Dataset (CAD-60)
  • UTKinect-Action3D (UTK)
  • Florence3D (F3D)
  • Kinect Activity Recognition Dataset (KARD)
  • MSR DailyActivity3D (MSR)

These datasets vary in the number of activities, subjects, and video lengths, providing a diverse testing ground.

Key Insights and Ideas

  • Kinetic Energy for Keyframes: Using kinetic energy to select keyframes is an effective way to capture the most dynamic and informative parts of an activity, reducing computational load and noise from static or repetitive frames.
  • Comprehensive Feature Extraction: Combining spatial, temporal, statistical, and orientation features provides a richer representation of human activities, leading to better discrimination compared to methods that focus on only one aspect.
  • Hybrid PSO for Robust Clustering: The combination of PSO's global search with Gaussian mutation for diversity and K-means for local refinement offers a powerful approach to overcome the limitations of individual algorithms, leading to more accurate and robust clustering.
  • Unsupervised Discovery is Crucial: The paper strongly emphasizes the need for unsupervised methods due to the impracticality of labeling large-scale, real-world activity data. Their approach directly addresses this gap.
  • PCA for Efficiency and Accuracy: Applying PCA before clustering not only reduces computational complexity but also improves clustering accuracy by focusing on the most discriminative features.

Key Results

The proposed HPGMK framework demonstrated superior performance across all five tested datasets.

  • Average Overall Accuracy: The method achieved average accuracies of 77.53% (CAD-60), 56.54% (UTK), 66.84% (F3D), 46.02% (KARD), and 40.12% (MSR).
  • Performance Improvement: The paper claims an average increase in accuracy of at least 4% compared to other state-of-the-art methods.
  • Robustness: HPGMK showed better performance on activities with high intra-class variation, corrupted data, and those performed in different poses (sitting/standing).
  • Component Impact: Experiments showed that combining all feature extraction methods (displacement, statistical, orientation) yielded the best discovery accuracy. The integration of Gaussian mutation and K-means significantly improved PSO's performance and convergence rate.
  • Clustering Time: The proposed method achieved competitive clustering times, benefiting from PCA and keyframe selection.

Limitations and Future Work

  • Pre-configured Number of Clusters: The current HPGMK algorithm requires the number of activities (clusters) to be pre-configured. The authors suggest extending the framework to automatically estimate this number.
  • Outlier Detection: The presence of outliers can negatively impact cluster centers. The paper suggests incorporating outlier detection methods to improve robustness.
  • Manual PSO Structure Setup: The manual tuning of PSO parameters and structure based on trial-and-error is time-consuming. Automating this setup process is identified as a potential future research direction to make the method more adaptable to various datasets and circumstances.

Summary updated

Abstract

Human activity discovery aims to cluster human activities without any prior knowledge of what defines each activity. However, most existing methods for human activity recognition are supervised, relying on labeled inputs for training. In reality, it is challenging to label human activity data due to its large volume and the diversity of human activities. To address this issue, this article proposes an unsupervised framework for human activity discovery in 3-D skeleton sequences. The framework includes a data preprocessing step that selects important frames based on kinetic energy and extracts relevant features, such as joint displacement, statistical displacement, angles, and orientation. To reduce the dimensionality of the extracted features, the framework uses principle component analysis. Unlike many other methods for human activity discovery, the proposed framework is fully unsupervised and does not rely on presegmented videos. To segment the time series of activities, the framework uses a sliding time window with some overlapping. The hybrid particle swarm optimization (PSO) with Gaussian mutation and K-means algorithm is then proposed to discover the activities. PSO is chosen for its powerful global search capability and simple implementation. To further improve the convergence rate of PSO, K-means is applied to the outcome centroids from each iteration of PSO. The experimental results on five datasets demonstrate that the proposed framework has superior performance in discovering activities compared to other state-of-the-art methods. The framework achieves an average increase in accuracy of at least 4%.
BibTeX
@article{hadikhani2023novel,
  author = {Parham Hadikhani and Daphne Teck Ching Lai and Wee-Hong Ong},
  title = {{A Novel Skeleton-Based Human Activity Discovery Using Particle Swarm Optimization With Gaussian Mutation}},
  journal = {IEEE Trans. Hum. Mach. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=tyuBiu6XY3},
}
Figure