MV-GMN: State Space Model for Multi-View Action Recognition

TL;DR

This paper introduces MV-GMN, a novel State Space Model (SSM) architecture that efficiently fuses multi-modal (RGB, skeleton), multi-view, and multi-temporal data for action recognition. By leveraging a Multi-View Graph Mamba network, MV-GMN achieves state-of-the-art performance while significantly reducing computational complexity compared to Transformer-based methods.

What Problem Does This Paper Address?

Action recognition, a crucial task in computer vision with applications ranging from surveillance to human-computer interaction, often requires understanding complex human movements from various perspectives and over time. While Transformer-based models have shown promise, they are computationally expensive, especially when dealing with multiple views and long temporal sequences. This paper addresses the challenge of achieving efficient and effective multi-view action recognition by proposing a new approach that reduces computational load without sacrificing accuracy. The core problem is the computational bottleneck of existing methods when integrating diverse data modalities and viewpoints for comprehensive action understanding.

Methods

The proposed MV-GMN (Multi-View Graph Mamba Network) is the central innovation. It's built upon the recent success of Mamba, an SSM-based model known for its linear time complexity in sequence processing. MV-GMN integrates Mamba with graph convolutional networks (GCNs) to handle the complexities of multi-view and multi-temporal data.

Key components include:

• Multi-Modal Feature Extraction: The framework uses pre-trained encoders (DeiT for RGB, Sparse-MSSTNet for skeletons) to extract features. A cross-attention mechanism is then employed to fuse these features, allowing skeletal data to guide the attention towards relevant RGB features.
• Bidirectional State Space Model Block: This block is the core of the Mamba integration. It introduces four scanning strategies (view-prioritized forward/backward, time-prioritized forward/backward) to effectively model temporal and spatial relationships across different views.
• Multi-View Graph Construction and Propagation: To capture relationships between different views and time instances, a graph is constructed. This graph uses both rule-based edges (time-consistent and view-consistent) and KNN-based edges to represent explicit and implicit connections between features. GCNs are then used to propagate information across this graph.
• MV-GMN Block: Each MV-GMN block combines a Bidirectional State Space Model Block and a GCN module, allowing for efficient feature aggregation across views and time.

Datasets

The paper evaluates MV-GMN on three widely used multi-modal action recognition datasets:

• NTU RGB+D 60: A benchmark with 56,880 video samples from 40 participants, covering 60 action classes. It includes depth maps, 3D skeletal, RGB, and infrared data.
• NTU RGB+D 120: An extension of NTU RGB+D 60, featuring 114,480 samples and 120 action classes, offering greater diversity.
• PKU-MMD: A concise multi-modality benchmark for 3D human action understanding, comprising over 2,000 video sequences and 5.4 million frames, recorded using Kinect v2 sensors.

Key Insights and Ideas

• SSMs for Multi-View Fusion: The paper pioneers the application of State Space Models (SSMs) for the challenging task of multi-view and multi-temporal action recognition. This moves beyond Transformer dominance and offers a more computationally efficient alternative.
• Graph-Mamba Synergy: The combination of graph convolutional networks (GCNs) for explicit relational modeling and Mamba for efficient sequential processing proves highly effective in capturing complex spatio-temporal dependencies across multiple views.
• Prioritizing View or Time: The introduction of distinct scanning strategies within the Bidirectional State Space Model Block (view-prioritized vs. time-prioritized) allows the model to adapt its focus, leading to improved performance depending on the specific action or data characteristics.
• KNN for Robust Graph Construction: Utilizing KNN-based edges in addition to rule-based ones allows the graph to capture more subtle, similarity-based relationships between features, enhancing the model's ability to learn from diverse viewpoints.

Key Results

MV-GMN demonstrates state-of-the-art performance across the evaluated datasets:

• NTU RGB+D 120: Achieved accuracies of 97.3% (cross-subject) and 96.7% (cross-view) with the MV-GMN (View-Time) configuration.
• NTU RGB+D 60: Achieved accuracies of 98.2% (cross-subject) and 99.7% (cross-view) with the MV-GMN (View-Time) configuration.
• PKU-MMD: Achieved accuracies of 98.0% (cross-subject) and 99.1% (cross-view) with the MV-GMN (View-Time) configuration.

Crucially, MV-GMN achieves these results with linear inference complexity, significantly outperforming Transformer-based baselines in terms of computational efficiency. The paper also shows that the MV-GMN (View-Prioritized) strategy generally outperforms the MV-GMN (Time-Prioritized) strategy.

Limitations and Future Work

• Hyperparameter Sensitivity: The paper notes that the performance is sensitive to hyperparameters like the number of KNN neighbors and MV-GMN blocks. Optimal values (e.g., 3 neighbors, 4 blocks) were identified through experimentation, suggesting that careful tuning is important.
• Future Work: While not explicitly detailed, the success of MV-GMN suggests future work could explore:
• Applying this architecture to other spatio-temporal tasks beyond action recognition.
• Investigating more sophisticated graph construction methods.
• Further optimizing the SSM and GCN components for even greater efficiency.
• Exploring different multi-modal fusion strategies.