TL;DR
This paper introduces the Multi-Modality Adaptive Feature Fusion Graph Convolutional Network (MMAFF) for skeleton-based action recognition. MMAFF enhances traditional graph convolutional networks by adaptively fusing multi-modal skeleton data (joints, bones, and limbs) and employing an attention mechanism to dynamically adjust receptive fields, leading to improved accuracy on challenging datasets.
What Problem Does This Paper Address?
Recognizing human actions from skeleton data is crucial for natural human-computer interaction and various applications. Skeleton data offers advantages over RGB video by being less susceptible to illumination and background changes. However, existing methods often struggle to effectively capture the complex spatio-temporal dynamics of human movement. Specifically, they often rely on fixed-size convolutional kernels, limiting their ability to adapt to different layers and datasets. Furthermore, traditional residual connections can lead to context loss when aggregating spatio-temporal information. This paper aims to address these limitations by proposing a more adaptive and robust framework for skeleton-based action recognition.
Methods
The core of this work is the Multi-Modality Adaptive Feature Fusion Framework (MMAFF). MMAFF processes skeleton data from multiple modalities: joints, bones, and a novel "limb" stream.
Key components include:
- Spatio-Temporal Adaptive Fusion (STAF) Block: This is the fundamental building block of MMAFF. Each STAF block comprises two sub-modules:
- Spatial Attention Graph Convolution (SAGC) Module: This module dynamically models spatial attention maps by using an attentional feature fusion mechanism to replace traditional residual connections. This helps in better aggregating spatial and temporal scale information.
- Temporal Adaptive Feature Fusion (TAFF) Module: This module is designed to dynamically adjust the size of convolution kernels and dilation rates at different network layers. This adaptive approach allows the network to capture more relevant features at different depths. It also incorporates an attention feature fusion mechanism to effectively integrate initial features with multi-scale aggregated features, addressing the problem of context aggregation and initial feature integration.
- Limb Stream: A novel contribution is the introduction of a "limb stream." Instead of processing joint and bone data independently, the limb stream integrates joint motion and bone motion data in the channel dimension. This approach is motivated by the observation that many actions involve coordinated limb movements, and treating limbs as a unified entity can lead to richer representations and reduced model complexity.
- Multi-Modality Fusion: The framework utilizes a three-way ensemble of the joint, bone, and limb streams. The final classification score is obtained by performing a weighted average of the individual stream predictions.
Datasets
The proposed MMAFF model was evaluated on two large-scale, widely used skeleton-based action recognition datasets:
- NTU-RGB+D 60: This dataset contains 56,880 sample data across 60 action classes, captured using three Kinect V2 cameras. It provides RGB, depth, and 3D skeletal data.
- NTU-RGB+D 120: This is an extension of NTU-RGB+D 60, featuring 113,945 samples across 120 action classes. It includes 32 different setups and captures data from three camera views.
The paper uses the standard cross-subject (X-Sub) and cross-view (X-View) or cross-setup (X-Set) benchmarks for evaluation on these datasets.
Key Insights and Ideas
- Adaptive Receptive Fields are Crucial: The paper argues that fixed receptive fields in temporal convolution are insufficient. Dynamically adjusting kernel sizes and dilation rates based on network layer depth is key to capturing context at different scales.
- Attention for Feature Fusion: Employing attention mechanisms for feature fusion, particularly to integrate initial features with higher-level temporal features, is more effective than traditional residual connections for preventing context loss.
- Limb-Centric Representation: Treating limb movements as a unified modality (the "limb stream") offers a more holistic representation for actions involving coordinated limb actions, potentially reducing computational overhead and improving performance.
- Multi-Modality Synergy: Combining information from joints, bones, and the novel limb stream through an ensemble approach leads to superior performance compared to single-modality methods.
Key Results
The MMAFF model achieved state-of-the-art results on both the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.
- On NTU-RGB+D 60, the proposed 3-ensemble approach achieved a top-1 accuracy of 93.1% (X-Sub) and 96.9% (X-View).
- On NTU-RGB+D 120, the 3-ensemble approach achieved a top-1 accuracy of 89.7% (X-Sub) and 91.2% (X-Set).
- Ablation studies demonstrated that the TAFF module (combining TA and TFF) improved baseline accuracy by up to 0.7% on NTU-RGB+D 60 and 0.8% on NTU-RGB+D 120.
- The limb stream alone outperformed traditional motion streams by 0.4% on NTU-RGB+D 60.
- The model also showed competitive convergence speeds compared to the CTR-GCN baseline.
Limitations and Future Work
- Feature Extraction Order: The authors acknowledge that extracting spatial information first and then temporal information might lead to feature redundancy. For future work, they plan to explore simultaneous extraction of features from both temporal and spatial graph dimensions.
- Computational Complexity: While the limb stream aims to reduce complexity, the introduction of the AFF module in the TAFF module resulted in an increase in the total number of parameters compared to the baseline.
- Language Integration: The authors suggest exploring the integration of language text models to further improve action recognition performance and reduce computational complexity.
Summary updated