What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition

Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu

TL;DR

This paper introduces a novel two-pronged approach to skeleton-based action recognition by modeling human poses from both a "semantic" (joint relationships) and a "spatial" (local joint configurations) perspective. By fusing these complementary views using specialized Transformer and sparse convolutional networks, the method achieves state-of-the-art performance on challenging action recognition benchmarks.

What Problem Does This Paper Address?

Human action recognition is a crucial task for applications like human-computer interaction and surveillance. While RGB data is rich but sensitive to environmental changes, skeleton data offers a more abstract and robust representation of human movement. Existing skeleton-based methods primarily rely on a "semantic perspective," where human joints are grouped based on their known anatomical relationships (e.g., connecting a hand to a wrist). This approach, while effective, can overlook important spatial configurations of joints that might be indicative of an action, especially when joints that are semantically distant are spatially close. The paper addresses this limitation by proposing a complementary "spatial perspective" that prioritizes local spatial arrangements of joints, regardless of their semantic labels, to identify discriminative patterns.

Methods

The core of the proposed method is a two-stream architecture that processes skeleton data from both semantic and spatial viewpoints.

  • Semantic Perspective (SEM-Net): This stream leverages a Transformer-like architecture, adapted for skeleton data. Key innovations include:
  • Cross-Frame Self-Attention (CFSA): A mechanism to effectively model both spatial and temporal dependencies between joints across frames, overcoming the limitations of purely intra-frame attention.
  • Global Regularization (GR): A technique to incorporate inherent, fixed relationships between specific joint pairs (e.g., left hand and right hand) into the attention mechanism, improving robustness.
  • Multi-Scale Temporal Feed-Forward (MS-TFF) Layer: An enhancement to the standard feed-forward layer that uses multiple dilation rates to capture temporal information over diverse window sizes, balancing performance and computational cost.
  • Spatial Perspective (SPA-Net): This stream treats skeleton data as sparse points in 3D space and uses sparse convolutional networks.
  • Sparse Data Transformation: Raw skeleton coordinates are converted into a sparse format suitable for efficient processing.
  • Sparse Convolutional Networks: Two types are proposed:
  • SPA-3+1D: Combines sparse 3D convolutions for spatial feature extraction with 1D convolutions for temporal modeling.
  • SPA-4D: Employs sparse 4D convolutions to directly capture spatio-temporal features, offering a more unified approach.

The outputs from both SEM-Net and SPA-Net are fused (averaged) to produce the final action prediction. The authors also incorporate bone information (joint-to-joint vectors) into the semantic stream for further improvement.

Datasets

The proposed method was evaluated on three widely used datasets for skeleton-based action recognition:

  • NTU-60: Contains 56,578 action clips across 60 classes, with 3D joint data from 25 joints. Evaluated on Cross-Subject (CS) and Cross-View (CV) benchmarks.
  • NTU-120: An extension of NTU-60, featuring 113,945 clips in 120 classes, captured by 106 volunteers across 32 camera setups. Evaluated on Cross-Subject (CS) and Cross-Setup (CE) benchmarks.
  • SHREC: A dataset for skeleton-based gesture recognition, providing 3D joint data of 22 hand joints. Evaluated on 14-class and 28-class gesture recognition tasks.

Key Insights and Ideas

  • Complementarity of Semantic and Spatial Views: The paper strongly argues that semantic and spatial perspectives are orthogonal and complementary. The semantic view captures explicit anatomical relationships, while the spatial view captures emergent local patterns, and fusing them leads to a more comprehensive understanding of actions.
  • Transformer for Skeleton Semantics: Adapting the Transformer architecture, successful in NLP, for modeling joint correlations in skeleton data is a key contribution. The CFSA and GR mechanisms are crucial for making this adaptation effective.
  • Sparse Convolutions for Spatial Patterns: Recognizing the sparsity of skeleton data in 3D space, the paper effectively utilizes sparse convolutions to avoid wasted computation and efficiently extract local spatial-temporal features.
  • SPA-4D vs. SPA-3+1D: The SPA-4D approach, directly modeling spatio-temporal features, generally outperforms the SPA-3+1D method, suggesting the benefit of a unified spatio-temporal convolution for sparse data.

Key Results

The proposed fused SEM-Net and SPA-Net model achieved state-of-the-art performance on all three tested datasets:

  • NTU-60: Achieved 92.6% accuracy on the CS benchmark and 97.1% on the CV benchmark.
  • NTU-120: Achieved 88.7% accuracy on the CS benchmark and 90.3% on the CE benchmark.
  • SHREC: Achieved 96.2% accuracy on the 14-class benchmark and 94.4% on the 28-class benchmark.

Ablation studies demonstrated the effectiveness of individual components, with the GR mechanism providing significant improvements for SEM-Net and the SPA-4D architecture proving superior for SPA-Net. The fusion of SEM-Net and SPA-Net consistently outperformed either network alone.

Limitations and Future Work

  • Fusion Strategy: The paper suggests that fusing the two perspectives at a later stage (e.g., averaging scores) might not be optimal. Future work could explore fusing the two perspectives earlier in the network for potentially greater synergy.
  • Unified Framework: The authors propose exploring whether the semantic and spatial perspectives can be unified into a single framework, rather than processed in parallel streams.
  • Generalizability: While tested on gesture recognition, further exploration of the model's generalizability to other types of human motion or different skeleton formats could be beneficial.

Summary updated

Abstract

Skeleton data, which consists of only the 2D/3D coordinates of the human joints, has been widely studied for human action recognition. Existing methods take the semantics as prior knowledge to group human joints and draw correlations according to their spatial locations, which we call the semantic perspective for skeleton modeling. In this paper, in contrast to previous approaches, we propose to model skeletons from a novel spatial perspective, from which the model takes the spatial location as prior knowledge to group human joints and mines the discriminative patterns of local areas in a hierarchical manner. The two perspectives are orthogonal and complementary to each other; and by fusing them in a unified framework, our method achieves a more comprehensive understanding of the skeleton data. Besides, we customized two networks for the two perspectives. From the semantic perspective, we propose a Transformer-like network that is expert in modeling joint correlations, and present three effective techniques to adapt it for skeleton data. From the spatial perspective, we transform the skeleton data into the sparse format for efficient feature extraction and present two types of sparse convolutional networks for sparse skeleton modeling. Extensive experiments are conducted on three challenging datasets for skeleton-based human action/gesture recognition, namely, NTU-60, NTU-120 and SHREC, where our method achieves state-of-the-art performance.
BibTeX
@misc{shi2020what,
  author = {Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu},
  title = {{What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition}},
  year = {2020},
  eprint = {2004.03259},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.03259},
}
Figure