Abstract
Few-shot skeleton-based action recognition is a challenging task due to the subtle differences between action categories and the temporal variability in human movement. We present PAINet (Parallel Attention Interaction Network), a novel framework that employs dual parallel attention branches to address spatial and temporal misalignments simultaneously. By integrating spatial cross-attention to isolate discriminative joint movements and temporal cross-attention to align asynchronous motion patterns, PAINet achieves state-of-the-art performance on the NTU RGB+D 120 and Kinetics benchmarks.
Introduction
Skeleton-based action recognition utilizes 3D joint coordinates to identify human activities, offering robustness against background noise. However, in few-shot settings—where only a handful of examples are available for a new category—models often struggle with two primary issues: similar spatial appearances and inconsistent temporal patterns. For instance, actions like "taking off glasses" and "taking off headphones" involve nearly identical hand-to-head movements, differing only in subtle joint positioning. Furthermore, the same action may be performed at varying speeds or start times across different clips.
Existing methods typically prioritize either intra-skeleton relations or inter-skeleton temporal alignment. PAINet overcomes these limitations by leveraging the complementarity of spatial and temporal attention in a parallel architecture.
Methodology
PAINet processes a query action against a support set through three main stages: Topology Encoding, Parallel Attention Branches, and Late Fusion.
Topology Encoding
To provide a strong inductive bias, the model enhances raw skeleton data using a Topology Encoding module. This includes a Joint-Part Relation component that groups joints into five major body parts (arms, legs, torso) and a Joint-Semantic Relation component that learns trainable biases to represent intrinsic connections between specific joints, such as the head and hands.
Parallel Attention Branches
The core of PAINet lies in its dual-pathway approach to feature alignment:
- Cross Spatial Alignment Branch: This branch employs spatial cross-attention between query and support sequences to highlight joints that distinguish one action from another. It utilizes a non-parametric Directional Average Symmetric Surface Metric (DASSM) to find optimal temporal matches between sequences.
- Cross Temporal Alignment Branch: This pathway first uses spatial self-attention to enrich individual skeleton features. It then applies a temporal cross-attention matcher to synchronize sequences that may be misaligned in time, effectively handling variations in action rhythm.
Late Fusion
The final classification is determined by summing the distances calculated by both branches. This fusion ensures that the model's prediction is grounded in both the spatial "what" (which joints move) and the temporal "when" (the rhythm of movement) of the action.
Experimental Results
PAINet was evaluated on the NTU RGB+D 120 and Kinetics datasets, consistently outperforming previous state-of-the-art (SOTA) methods across various Graph Convolutional Network (GCN) backbones:
- NTU-T (1-shot): Achieved 82.4% accuracy, a significant improvement over the previous best of 75.1%.
- NTU-S (1-shot): Reached 84.6% accuracy.
- Kinetics (1-shot): Achieved 42.4% accuracy. While lower due to noisy 2D-to-3D skeleton estimation, it remains superior to existing few-shot methods on this benchmark.
Ablation studies confirmed that both the spatial cross-attention and the temporal matcher are essential, as removing either component resulted in a measurable performance decline.
Conclusion
PAINet demonstrates that parallel spatial and temporal alignment is a powerful strategy for few-shot skeleton-based action recognition. By treating topology as a prior and utilizing cross-attention to bridge the gap between query and support samples, the model effectively handles the nuances of human activity. Future work will focus on improving computational efficiency and exploring multi-modal integration to enhance robustness in noisy environments.
Summary updated