Research on Multi-Scale Spatio-Temporal Graph Convolutional Human Behavior Recognition Method Incorporating Multi-Granularity Features

Yulin Wang, Tao Song, Yichen Yang, Hong ZhengSensors 2024 · DOI

TL;DR

This paper introduces a novel multi-scale spatio-temporal graph convolutional network (MS-TGCN) that leverages multi-granularity skeleton features to improve human behavior recognition, especially for similar actions. By introducing a fine-grained skeleton partitioning strategy and an adaptive cross-scale feature fusion layer, the model can better distinguish subtle differences in human movements, achieving state-of-the-art results on the MSR Action 3D dataset.

What Problem Does This Paper Address?

Recognizing human actions from skeleton data is a crucial task in computer vision with applications in surveillance, human-computer interaction, and more. Existing methods, particularly those based on graph convolutional networks (GCNs), often struggle with two main issues:

  1. Insensitivity to Local Movements: GCNs can sometimes overemphasize global structural information, making them less sensitive to subtle, localized movements that are critical for distinguishing between similar actions (e.g., different types of gestures).
  2. Bias Weighting in Spatial Modeling: When modeling spatial relationships between joints using high-order adjacency matrices, joints closer to a target joint tend to dominate the recognition process. This "bias weighting" can lead to overlooking important information from more distant joints, hindering the recognition of actions that involve coordinated movements across the entire body.

This paper aims to address these limitations by developing a method that can capture both global and fine-grained spatial-temporal features, leading to more robust and accurate recognition, especially for actions that are easily confused.

Methods

The proposed approach, termed Multi-Scale Spatio-Temporal Graph Convolutional Network (MS-TGCN) incorporating Multi-Granularity Features, introduces several key innovations:

  • Multi-Granularity Skeleton Partitioning: The core idea is to represent human skeletons at different levels of detail. The original skeleton data is partitioned into streams of varying granularities. This is achieved by creating "fine-grained" skeletons that add more joints (e.g., from 20 to 23 or 25 joints) by averaging adjacent joints in the coarse-grained skeleton. This allows the model to capture more nuanced spatial information.
  • Adaptive Cross-Scale Feature Fusion Layer (CSFL): To effectively combine the information from different granularities, an adaptive CSFL is proposed. This layer uses a normalized Gaussian function to learn a feature-mapping relationship between coarse and fine-grained skeleton representations. It then fuses these features, guiding the model to focus on discriminative representations that differentiate similar behaviors.
  • Sparse Multi-Scale Adjacency Matrix: To combat the bias weighting issue in spatial modeling, the paper proposes a method of subtracting K-order adjacency matrices from (K-1)-order matrices. This creates a sparse adjacency matrix that aims to assign more balanced weights to joints at different distances, allowing the model to better capture relationships between distant joints.
  • End-to-End MS-TGCN Architecture: These components are integrated into an end-to-end GCN framework. The MS-TGCN block itself utilizes multi-scale adjacency matrices for spatial feature extraction and multi-scale temporal convolutional networks (MS-TCN) for temporal feature extraction.

Datasets

The primary dataset used for evaluation is the MSR Action 3D dataset. This dataset consists of 3D skeleton data for 20 human actions performed by 10 subjects. It is noted for containing several highly similar actions, making it a challenging benchmark for evaluating fine-grained action recognition. The authors employ a subject-based cross-validation strategy for evaluation.

Key Insights and Ideas

  • Granularity Matters: The paper strongly suggests that representing human skeletons at multiple levels of detail is crucial for distinguishing similar actions. Fine-grained details, which might be lost in coarse representations, can hold the key to differentiating subtle movements.
  • Fusion is Key: Simply having multi-granularity data isn't enough; effectively fusing these different representations is essential. The CSFL's adaptive nature allows it to dynamically learn how to blend information from various skeleton resolutions.
  • Beyond Local Joints: The bias weighting problem in traditional GCNs is a real hurdle. By introducing a sparse multi-scale adjacency matrix, the authors demonstrate a way to mitigate this, enabling the model to consider a broader range of joint interactions.
  • Similar Actions are Hard: The MSR Action 3D dataset's inherent similarity between actions highlights the need for methods that can capture minute differences, which this paper directly addresses.

Key Results

The proposed MS-TGCN method achieved a state-of-the-art accuracy of 95.67% on the MSR Action 3D dataset. This represents a significant improvement over existing methods. Specifically:

  • The introduction of the multi-scale adjacency matrix improved accuracy by up to 2.76% at K=6.
  • Fusing multi-granularity features (specifically, using a combination of 20, 23, and 25 joints with a proportional parameter λ=0.1) led to a 0.79% increase in accuracy compared to using only 20 joints.
  • The method showed particular improvement in distinguishing actions like "drawing a fork," "drawing a circle," and "bending down." For instance, the accuracy for "bending down" increased from 58.3% to 77.8% when using 23 or 25 joints.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Population Variability: The model's performance might be affected by variations in skeletal spatial scales across different populations (e.g., adults vs. children). Normalizing skeletal spatial structures is identified as an area for future work.
  • Action Speed: The speed at which an action is performed can significantly alter its semantic meaning. The current model might not fully capture these speed-dependent nuances, suggesting that temporal fine-grained aspects of actions warrant further investigation.
  • Joint Insertion Strategy: While adding joints at wrists, waist, and lower legs improved recognition for some actions, it led to a decrease in accuracy for others (e.g., those heavily reliant on finger and arm movements). This suggests that the strategy for adding fine-grained joints could be further refined to be more action-specific.

Summary updated

Abstract

Aiming at the problem that the existing human skeleton behavior recognition methods are insensitive to human local movements and show inaccurate recognition in distinguishing similar behaviors, a multi-scale spatio-temporal graph convolution method incorporating multi-granularity features is proposed for human behavior recognition. Firstly, a skeleton fine-grained partitioning strategy is proposed, which initializes the skeleton data into data streams of different granularities. An adaptive cross-scale feature fusion layer is designed using a normalized Gaussian function to perform feature fusion among different granularities, guiding the model to focus on discriminative feature representations among similar behaviors through fine-grained features. Secondly, a sparse multi-scale adjacency matrix is introduced to solve the bias weighting problem that amplifies the multi-scale spatial domain modeling process under multi-granularity conditions. Finally, an end-to-end graph convolutional neural network is constructed to improve the feature expression ability of spatio-temporal receptive field information and enhance the robustness of recognition between similar behaviors. The feasibility of the proposed algorithm was verified on the public behavior recognition dataset MSR Action 3D, with a accuracy of 95.67%, which is superior to existing behavior recognition methods.
BibTeX
@inproceedings{wang2024research,
  author = {Yulin Wang and Tao Song and Yichen Yang and Hong Zheng},
  title = {{Research on Multi-Scale Spatio-Temporal Graph Convolutional Human Behavior Recognition Method Incorporating Multi-Granularity Features}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24237595},
}
Figure