Spatial-Temporal Self-Attention Enhanced Graph Convolutional Networks for Fitness Yoga Action Recognition

Guixiang Wei, Huijian Zhou, Liping Zhang, Jianji WangSensors 2023 · DOI

TL;DR

This paper introduces STSAE-GCN, a novel skeleton-based approach for recognizing fitness yoga actions. By incorporating a spatial-temporal self-attention module (STSAM), the model significantly enhances its ability to capture complex pose dynamics, achieving state-of-the-art accuracy on a newly created yoga dataset.

What Problem Does This Paper Address?

Fitness yoga is gaining popularity, and while tools like Microsoft Kinect have been used for guidance, they are often expensive and inconvenient. Existing human action recognition methods, especially those relying on RGB data, struggle with varying backgrounds and lighting conditions. Skeleton-based methods offer a more robust alternative by focusing on joint movements, but accurately recognizing nuanced actions like yoga poses, which involve subtle spatial and temporal variations, remains a challenge. The authors aim to develop a system that can accurately recognize fitness yoga actions from readily available RGB video (e.g., from smartphones), making it accessible for independent learning and practice.

Methods

The core of the proposed method is the Spatial-Temporal Self-Attention Enhanced Graph Convolutional Network (STSAE-GCN). This architecture builds upon Graph Convolutional Networks (GCNs), which are well-suited for representing human skeletons as graphs (joints as nodes, bones as edges).

Key components include:

  • Adaptive Graph Convolutional Networks (AGCN): Instead of using a fixed adjacency matrix representing natural human skeleton connections, the AGCN learns an adaptive adjacency matrix. This allows the model to better capture specific correlations between joints that are crucial for action recognition, even if they aren't directly connected in a standard skeleton.
  • Multi-Branch Temporal Convolutional Networks (MTCN): To capture temporal dynamics, the authors employ an MTCN. This network uses multiple parallel branches with different temporal receptive fields (via dilated convolutions) to extract features at various time scales, aiming for a more comprehensive understanding of motion over time.
  • Spatial-Temporal Self-Attention Module (STSAM): This is the novel contribution. Inspired by self-attention mechanisms in NLP and computer vision, the STSAM enhances the model's ability to focus on the most relevant spatial and temporal features. It generates spatial and temporal attention maps that are integrated back into the feature representations, allowing the model to dynamically weigh important joint relationships and temporal sequences. The STSAM is designed to be "plug-and-play," meaning it can be added to existing GCN-based action recognition models.

Datasets

  • Yoga10: A new dataset specifically created for this research. It comprises 960 video clips of 32 volunteers performing 10 distinct fitness yoga actions. The videos were captured from three different angles simultaneously. The dataset is split into 150 clips for training and 810 for evaluation. Raw RGB videos were used, and pose estimation (using HRNet) was applied to extract skeleton data.

Key Insights and Ideas

  • Learned Skeleton Connectivity is Crucial: The success of the AGCN highlights that fixed skeleton connectivity is insufficient for nuanced actions like yoga. Learning adaptive relationships between joints significantly improves feature extraction.
  • Multi-Scale Temporal Analysis is Key: The MTCN's multiple branches for temporal feature extraction demonstrate that understanding motion at different time scales is vital for accurate action recognition.
  • Self-Attention for Dynamic Focus: The STSAM's ability to dynamically weigh spatial and temporal information is a powerful mechanism for capturing the subtle, evolving dynamics of yoga poses. This attention mechanism is shown to be effective and generalizable.
  • Plug-and-Play Attention for Model Improvement: The STSAM's effectiveness when applied to other existing GCN-based models (AAGCN, MSG3D, CTRGCN) underscores its value as a general enhancement module for skeleton-based action recognition.

Key Results

  • State-of-the-Art Performance: The proposed STSAE-GCN achieved a top-1 accuracy of 93.83% on the Yoga10 dataset, outperforming existing state-of-the-art methods.
  • STSAM Effectiveness: Ablation studies showed that combining both spatial and temporal self-attention (STSAM) yielded the best results, outperforming models with only spatial (SSAM) or only temporal (TSAM) attention.
  • Parallel Attention is Superior: Placing the spatial and temporal attention modules in parallel within the STSAM module achieved better performance than placing them in series.
  • Robustness to Viewpoint Variation: While view-3 showed reduced accuracy due to pose estimation issues, the model demonstrated good performance across different views, with view-2 yielding the highest accuracy. The authors suggest this view might be optimal for learning.
  • Real-time Performance: The model achieved an average recognition time of 0.02 seconds per skeleton clip, meeting real-time requirements for practical applications.

Limitations and Future Work

  • Pose Estimation Dependency: The accuracy of the model is inherently dependent on the quality of the pose estimation algorithm. Inaccurate pose estimation, particularly from challenging views (like view-3) or with significant occlusions, can degrade performance.
  • Dataset Specificity: While Yoga10 is a valuable contribution, future work could involve larger and more diverse yoga datasets to further improve generalization.
  • Occlusion Handling: Although the authors claim robustness to occlusion, severe occlusions can still impact pose estimation and thus recognition accuracy. Further research into more robust occlusion handling mechanisms could be beneficial.
  • Practical Application Guidance: The authors suggest that in practical applications, displaying the recognized action and highlighting discrepancies could help users correct their poses. This hints at future work in interactive feedback systems.

Summary updated

BibTeX
@article{wei2023spatial,
  author = {Guixiang Wei and Huijian Zhou and Liping Zhang and Jianji Wang},
  title = {{Spatial-Temporal Self-Attention Enhanced Graph Convolutional Networks for Fitness Yoga Action Recognition}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23104741},
}
Figure