Skeleton-Based Action Recognition with Synchronous Local and Non-Local Spatio-Temporal Learning and Frequency Attention

Guyue Hu, Bo Cui, Shan Yu

TL;DR

This paper introduces a novel approach for skeleton-based action recognition by simultaneously learning local and non-local spatio-temporal features and incorporating frequency domain information. Their method, SLnL-rFA, achieves state-of-the-art performance on large-scale datasets by effectively capturing both fine-grained details and long-range semantics, and by leveraging frequency patterns often overlooked in prior work.

What Problem Does This Paper Address?

Skeleton-based action recognition aims to understand human actions from sequences of joint positions. While existing methods often use hierarchical local networks (like CNNs or GCNs) to extract spatio-temporal features, they face two main challenges:

  1. Asynchronous Feature Capture: Local operations tend to capture detailed information in lower layers and semantic information in higher layers, leading to an asynchronous fusion of these crucial aspects of action dynamics.
  2. Ignoring Frequency Domain: Many human actions, such as shaking hands or brushing teeth, exhibit distinct periodic patterns in the frequency domain. However, most current methods are confined to the spatio-temporal domain, neglecting this valuable information.

This paper seeks to bridge this gap by proposing a method that synchronously captures local and non-local spatio-temporal information and explicitly incorporates frequency domain analysis.

Methods

The proposed method, SLnL-rFA, consists of several key components:

  • Adaptive Transform Network: This initial stage augments and transforms the skeletal data, potentially creating richer representations by considering multiple coordinate systems and joint arrangements.
  • Residual Frequency Attention (rFA) Block: This novel block addresses the limitation of ignoring the frequency domain. It transforms the spatio-temporal data into the frequency domain (using 2D FFT) and applies attention mechanisms to selectively focus on discriminative frequency patterns (sinusoidal and cosine components). A residual connection is then used to combine these frequency-aware features with the original spatio-temporal information.
  • Synchronous Local and Non-Local (SLnL) Block: This block is designed to overcome the asynchronous capture of local and non-local dependencies. It integrates multiple non-local modules (temporal, spatial, and spatio-temporal) in parallel with local operations (convolutional layers) within the same layer. This allows for the simultaneous extraction and fusion of both fine-grained local details and long-range semantic information.
  • Soft-Margin Focal Loss (SMFL): To optimize the learning process, a new loss function is introduced. SMFL combines the benefits of focal loss (adaptive data selection, focusing on hard samples) with a soft-margin mechanism. This encourages intrinsic margins in classifiers, improving class separability and robustness.

The overall architecture processes both position and velocity information of the skeleton, feeding them through these blocks and ultimately combining their outputs for a pseudo multi-task learning objective.

Datasets

The paper evaluates its method on two large-scale datasets:

  • NTU RGB+D: This is described as the largest in-door action recognition dataset, containing 56,000 clips across 60 actions performed by 40 subjects. The evaluation uses standard cross-subject (CS) and cross-view (CV) protocols.
  • Kinetics: This is presented as the largest unconstrained action recognition dataset, with 300,000 video clips in 400 classes. The skeleton data is estimated using OpenPose.

Key Insights and Ideas

  • Frequency Domain is Crucial: The paper highlights that actions often have inherent frequency patterns that are missed by purely spatio-temporal models. Their rFA block effectively leverages this information.
  • Synchronous Learning is Key: By integrating local and non-local operations within the same layers (SLnL block), the model can simultaneously capture and fuse detailed and semantic information, leading to a more comprehensive understanding of action dynamics.
  • Loss Function Matters: The proposed SMFL not only helps in selecting informative samples during training but also enforces better separation between classes, contributing to improved performance.
  • Early Fusion is Beneficial: The experiments suggest that incorporating non-local information in earlier layers (as done by SLnL) is more effective than relying on it only in deeper layers, as is common in some prior work.

Key Results

The proposed SLnL-rFA model significantly outperforms state-of-the-art methods on both NTU RGB+D and Kinetics datasets.

  • NTU RGB+D: Achieved 89.1% accuracy on the cross-subject (CS) protocol and 94.9% on the cross-view (CV) protocol, outperforming previous best methods by substantial margins (e.g., 2.6% and 3.8% respectively over HCN).
  • Kinetics: Achieved 36.6% top-1 accuracy and 59.1% top-5 accuracy, outperforming ST-GCN by 5.9% (top1) and 6.3% (top5).
  • Ablation Studies: Demonstrated the effectiveness of each component: rFA improved performance over a baseline without frequency attention, and the SLnL block with synchronous learning provided significant gains over methods that only used temporal or spatial non-local blocks. The SMFL also showed superior performance compared to CE, FL, and SMCE.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the inclusion of non-local modules and frequency transformations can increase computational complexity.
  • Future Work: The paper implicitly suggests future work could involve exploring more sophisticated transform networks, further optimizing the fusion of multi-domain information, and potentially applying this framework to other related tasks. The effectiveness of deeper SLnL blocks suggests exploring optimal configurations for different network depths.

Summary updated

Abstract

Benefiting from its succinctness and robustness, skeleton-based action recognition has recently attracted much attention. Most existing methods utilize local networks (e.g recurrent, convolutional, and graph convolutional networks) to extract spatio-temporal dynamics hierarchically. As a consequence, the local and non-local dependencies, which contain more details and semantics respectively, are asynchronously captured in different level of layers. Moreover, existing methods are limited to the spatio-temporal domain and ignore information in the frequency domain. To better extract synchronous detailed and semantic information from multi-domains, we propose a residual frequency attention (rFA) block to focus on discriminative patterns in the frequency domain, and a synchronous local and non-local (SLnL) block to simultaneously capture the details and semantics in the spatio-temporal domain. Besides, a soft-margin focal loss (SMFL) is proposed to optimize the learning whole process, which automatically conducts data selection and encourages intrinsic margins in classifiers. Our approach significantly outperforms other state-of-the-art methods on several large-scale datasets.
BibTeX
@inproceedings{hu2019skeleton,
  author = {Guyue Hu and Bo Cui and Shan Yu},
  title = {{Skeleton-Based Action Recognition with Synchronous Local and Non-Local Spatio-Temporal Learning and Frequency Attention}},
  booktitle = {ICME},
  year = {2019},
  url = {https://openreview.net/forum?id=sIU8ogfxNiS},
}
Figure