TL;DR
This paper introduces CA-MambaPose, a novel framework for 3D human pose estimation that leverages both RGB images and event streams to capture richer spatial and temporal context. By employing a Mamba-based fusion module and a Pose-Context Feature Exchange mechanism, CA-MambaPose achieves state-of-the-art results on challenging benchmarks while maintaining computational efficiency.
What Problem Does This Paper Address?
Estimating the 3D pose of humans from images or videos is a fundamental yet challenging task in computer vision, with applications ranging from action recognition to human-robot interaction. While many existing methods rely on lifting 2D pose sequences to 3D, they often struggle in complex scenarios due to the limited spatial context provided by 2D joint coordinates alone. This can lead to ambiguity, especially in cases of occlusion or depth uncertainty. Furthermore, relying solely on RGB information might miss crucial temporal dynamics. Existing methods that attempt to incorporate temporal information can be computationally expensive. This paper addresses the need for a more robust and efficient 3D human pose estimation method that can effectively utilize both spatial and temporal context from multiple modalities.
Methods
The core of CA-MambaPose lies in its innovative approach to fusing information from RGB images and event streams. The framework starts by using a pretrained 2D pose detector to extract multi-scale feature maps from both RGB and event inputs. These features are then processed by a Deformable Context Extraction module, which uses deformable attention to capture joint-centric spatial and temporal contextual cues, guided by the detected 2D joints.
The key novelty is the Cross Modality Fusion Mamba (CMFM) module. Unlike traditional attention mechanisms with quadratic complexity, CMFM utilizes Mamba's linear-time state space models (SSMs) to fuse the RGB and event features in a hidden state space. This allows for efficient capture of spatio-temporal relationships and reduces domain gaps between modalities. CMFM incorporates bidirectional SS2D (2D-Selective-Scan) layers and an Efficient Channel Attention (ECA) module to enhance feature representation.
To further bridge the gap between contextual features and 2D pose embeddings, the Pose-Context Feature Exchange module is introduced. This module uses SSMs to learn a joint representation, promoting message-passing between positional information from 2D joints and the contextual cues from fused features. This approach is more computationally efficient than transformer-based methods.
Finally, an Inter-joint Relation Modeling module, inspired by PoseFormer, uses a spatial transformer encoder to model global skeleton structure and dependencies. The output is then fed to a regression head for final 3D pose estimation.
Datasets
The paper evaluates CA-MambaPose on two widely-used video-based 3D human pose estimation benchmarks:
- Human3.6M: A large-scale indoor benchmark for single-person 3D pose estimation.
- MPI-INF-3DHP: A dataset gathered in diverse settings (green screen, non-green screen, outdoor) for multi-person 3D pose estimation.
The event stream used in the experiments is generated offline from RGB images using the v2e toolbox.
Key Insights and Ideas
- Event Streams for 3D HPE: The paper demonstrates the significant benefit of incorporating event streams, which capture high-dynamic range and precise temporal information, for improving 3D human pose estimation, especially in challenging scenarios.
- Mamba for Cross-Modality Fusion: Leveraging Mamba's SSM architecture for fusing features from different modalities (RGB and event) offers a computationally efficient alternative to attention-based fusion, achieving strong performance with linear complexity.
- Context is Crucial: The framework emphasizes the importance of "context-awareness" by explicitly extracting and fusing spatial and temporal context from multi-scale feature maps, going beyond simple 2D joint coordinates.
- Pose-Context Feature Exchange: The novel module effectively integrates 2D pose information with rich contextual features, further refining the representation for each joint.
Key Results
CA-MambaPose achieves state-of-the-art results on both Human3.6M and MPI-INF-3DHP datasets.
- On Human3.6M, the single-frame CA-MambaPose achieved an MPJPE of 36.5mm and a PA-MPJPE of 28.6mm, outperforming previous transformer-based methods that often use more frames.
- On MPI-INF-3DHP, the method achieved an AUC of 84.0% and a P1 error of 20.2mm, also setting new state-of-the-art records.
- The paper highlights that incorporating event streams and using the Mamba-based fusion module leads to significant error reductions (e.g., from 51.2mm to 36.5mm MPJPE on Human3.6M) without a substantial increase in computational cost or model parameters.
Limitations and Future Work
The authors acknowledge that for all single-frame methods, including CA-MambaPose, mitigating jitters in pose estimation remains a challenge compared to multi-frame methods. They suggest that a promising future direction is to extend their method to model short-term temporal dependencies, potentially by incorporating a Temporal Transformer to handle temporal correlations of each joint across frames independently, without incurring excessively high computational costs.
Summary updated