TL;DR
This paper introduces the Merge-and-Split Graph Convolutional Network (MS-GCN) for skeleton-based human interaction recognition. MS-GCN effectively captures crucial correlation features between body parts, which are often missed by traditional methods, by treating interactions as a global problem and employing a novel "Merge-and-Split" graph structure. This approach achieves state-of-the-art performance on benchmark datasets.
What Problem Does This Paper Address?
Human interaction recognition is a vital area for applications like human-robot interaction. While skeleton-based methods are robust to lighting and occlusion, they often struggle to capture the nuanced correlations between different body parts during an interaction. Traditional approaches tend to treat joints as independent features or model interactions simplistically, leading to difficulties in distinguishing subtle differences between similar actions (e.g., "handshake" vs. "high five"). Existing graph convolutional networks (GCNs) often treat individuals in an interaction as separate entities, overlooking the critical interdependencies. This paper aims to bridge this gap by developing a model that can effectively extract and leverage these complex, multi-part correlations within human interactions.
Methods
The core of the proposed MS-GCN lies in its novel Merge-and-Split Graph structure. This structure addresses the limitations of traditional GCNs by:
- Merging: It unifies the joint information from two individuals into a single feature space.
- Splitting: It then divides this unified graph into multiple "hierarchical sets" based on semantic relationships (e.g., grouping hand joints together). This allows for focused feature extraction within these semantically related groups.
To further enhance interaction recognition, the MS-GCN incorporates two additional modules:
- Short-term Dependence (STD) Module: This module is designed to capture fine-grained temporal and motion characteristics of individual joints and their short-term variations. This is crucial for distinguishing subtle differences in actions that might be missed by models focusing only on long-term trends.
- Hierarchical Guided Attention (HGA) Module: This module intelligently assigns weights to the different hierarchical sets identified by the Merge-and-Split graph. It highlights the sets that are most relevant to the specific interaction being recognized, thereby focusing the model's attention on the most informative features.
The Merge-and-Split Graph Convolution (MSGC) module combines the novel graph structure with GCN principles to extract rich semantic information between adjacent joint points within these hierarchical sets.
Datasets
The proposed MS-GCN was evaluated on two widely recognized benchmark datasets for skeleton-based human interaction recognition:
- NTU-60 (NTU RGB+D): This dataset contains 40,320 action clips from 60 action classes, performed by 25 actors from three viewpoints. It includes 11 two-person interaction categories.
- NTU-120 (NTU RGB+D 120): This is an extension of NTU-60, featuring 118,081 action clips from 120 action classes and 26 two-person interaction categories. It offers greater diversity in poses, actions, and environmental conditions.
The paper also mentions evaluating the model on single-person action recognition tasks using the X-Sub split of NTU-120.
Key Insights and Ideas
- Interactions are Global: The paper argues that treating interactions as a global problem, rather than just a collection of individual actions, is key to better recognition.
- Semantic Grouping is Powerful: Splitting the interaction graph into semantically meaningful "hierarchical sets" allows the model to focus on relevant body part correlations.
- Short-term Dynamics Matter: Capturing short-term motion variations is critical for distinguishing subtle differences between similar interaction types.
- Attention Guides Focus: The HGA module effectively learns to prioritize information from the most relevant hierarchical sets, improving efficiency and accuracy.
- Merge-and-Split Graph's Advantage: The novel graph structure itself provides a significant improvement by explicitly modeling inter-person and intra-person joint relationships within a unified framework.
Key Results
The MS-GCN achieved state-of-the-art performance on both the NTU-60 and NTU-120 datasets. Specifically:
- NTU-120: The MS-GCN (joint + bone fusion) achieved an accuracy of 92.52% on the X-Set, outperforming previous methods.
- NTU-60: The MS-GCN (joint + bone fusion) achieved an accuracy of 98.86% on the X-View.
- Ablation Studies: The paper demonstrated the effectiveness of each component:
- The Merge-and-Split graph alone improved accuracy by 1.01% over a single-splitting graph.
- Adding the STD module further boosted performance.
- Adding the HGA module also led to significant improvements.
- The full MS-GCN model (MSGC + STD + HGA) achieved an overall improvement of 1.92% compared to the baseline.
- Comparison with CTR-GCN: The MS-GCN showed higher accuracy than CTR-GCN in 71 out of 120 action classes on NTU-120, particularly for interaction behaviors.
Limitations and Future Work
The authors acknowledge two main limitations:
- Complex Motion Behaviors: For highly complex motion behaviors, the MS-GCN might need further refinement to extract even more critical correlation features.
- Scope of Interaction: Currently, the MS-GCN is designed for single and two-person interactions. The authors suggest future work should focus on extending the model to handle multi-person sports and more complex group interactions.
The paper also implies that future work could explore integrating more modalities or further optimizing the graph construction and attention mechanisms.
Summary updated