Parallel Attention Interaction Network for Few-Shot Skeleton-based Action Recognition

Xingyu Liu, Sanping Zhou, Le Wang, Gang Hua

Abstract

Few-shot skeleton-based action recognition is a challenging task due to the subtle differences between action categories and the temporal variability in human movement. We present PAINet (Parallel Attention Interaction Network), a novel framework that employs dual parallel attention branches to address spatial and temporal misalignments simultaneously. By integrating spatial cross-attention to isolate discriminative joint movements and temporal cross-attention to align asynchronous motion patterns, PAINet achieves state-of-the-art performance on the NTU RGB+D 120 and Kinetics benchmarks.

Introduction

Skeleton-based action recognition utilizes 3D joint coordinates to identify human activities, offering robustness against background noise. However, in few-shot settings—where only a handful of examples are available for a new category—models often struggle with two primary issues: similar spatial appearances and inconsistent temporal patterns. For instance, actions like "taking off glasses" and "taking off headphones" involve nearly identical hand-to-head movements, differing only in subtle joint positioning. Furthermore, the same action may be performed at varying speeds or start times across different clips.

Figure 1: Comparison of Cross Attention and Self Attention mechanisms in identifying discriminative joints for similar actions.

Existing methods typically prioritize either intra-skeleton relations or inter-skeleton temporal alignment. PAINet overcomes these limitations by leveraging the complementarity of spatial and temporal attention in a parallel architecture.

Methodology

PAINet processes a query action against a support set through three main stages: Topology Encoding, Parallel Attention Branches, and Late Fusion.

Topology Encoding

To provide a strong inductive bias, the model enhances raw skeleton data using a Topology Encoding module. This includes a Joint-Part Relation component that groups joints into five major body parts (arms, legs, torso) and a Joint-Semantic Relation component that learns trainable biases to represent intrinsic connections between specific joints, such as the head and hands.

Parallel Attention Branches

The core of PAINet lies in its dual-pathway approach to feature alignment:

  1. Cross Spatial Alignment Branch: This branch employs spatial cross-attention between query and support sequences to highlight joints that distinguish one action from another. It utilizes a non-parametric Directional Average Symmetric Surface Metric (DASSM) to find optimal temporal matches between sequences.
  2. Cross Temporal Alignment Branch: This pathway first uses spatial self-attention to enrich individual skeleton features. It then applies a temporal cross-attention matcher to synchronize sequences that may be misaligned in time, effectively handling variations in action rhythm.
Figure 2: The overall architecture of PAINet, showcasing the parallel spatial and temporal alignment pathways.

Late Fusion

The final classification is determined by summing the distances calculated by both branches. This fusion ensures that the model's prediction is grounded in both the spatial "what" (which joints move) and the temporal "when" (the rhythm of movement) of the action.

Experimental Results

PAINet was evaluated on the NTU RGB+D 120 and Kinetics datasets, consistently outperforming previous state-of-the-art (SOTA) methods across various Graph Convolutional Network (GCN) backbones:

  • NTU-T (1-shot): Achieved 82.4% accuracy, a significant improvement over the previous best of 75.1%.
  • NTU-S (1-shot): Reached 84.6% accuracy.
  • Kinetics (1-shot): Achieved 42.4% accuracy. While lower due to noisy 2D-to-3D skeleton estimation, it remains superior to existing few-shot methods on this benchmark.

Ablation studies confirmed that both the spatial cross-attention and the temporal matcher are essential, as removing either component resulted in a measurable performance decline.

Conclusion

PAINet demonstrates that parallel spatial and temporal alignment is a powerful strategy for few-shot skeleton-based action recognition. By treating topology as a prior and utilizing cross-attention to bridge the gap between query and support samples, the model effectively handles the nuances of human activity. Future work will focus on improving computational efficiency and exploring multi-modal integration to enhance robustness in noisy environments.

Summary updated

Abstract

Learning discriminative features from very few labeled samples to identify novel classes has received increasing attention in skeleton-based action recognition. Existing works aim to learn action-specific embeddings by exploiting either intra-skeleton or inter-skeleton spatial associations, which may lead to less discriminative representations. To address these issues, we propose a novel Parallel Attention Interaction Network (PAINet) that incorporates two complementary branches to strengthen the match by inter-skeleton and intraskeleton correlation. Specifically, a topology encoding module utilizing topology and physical information is proposed to enhance the modeling of interactive parts and joint pairs in both branches. In the Cross Spatial Alignment branch, we employ a spatial cross-attention module to establish joint associations across sequences, and a directional Average Symmetric Surface Metric is introduced to locate the closest temporal similarity. In parallel, the Cross Temporal Alignment branch incorporates a spatial self-attention module to aggregate spatial context within sequences as well as applies the temporal cross-attention network to correct misalignment temporally and calculate similarity. Extensive experiments on three skeleton benchmarks, namely NTU-T, NTU-S, and Kinetics, demonstrate the superiority of our framework and consistently outperform state-of-the-art methods.
BibTeX
@inproceedings{liu2023parallel,
  author = {Xingyu Liu and Sanping Zhou and Le Wang and Gang Hua},
  title = {{Parallel Attention Interaction Network for Few-Shot Skeleton-based Action Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00133},
}
Figure