TL;DR
This paper introduces HOIMamba, a novel Human-Object Interaction (HOI) detection framework that leverages the Mamba architecture to efficiently disentangle and progressively learn interaction semantics. By combining the strengths of single, two, and three-branch HOI detection methods and introducing a novel progressive learning strategy, HOIMamba achieves state-of-the-art performance, particularly on challenging rare interaction categories.
What Problem Does This Paper Address?
Human-Object Interaction (HOI) detection aims to identify not only the presence of humans and objects in an image but also the specific action or relationship between them, typically represented as (human, verb, object) triplets. Existing HOI detection methods face a trade-off between efficiency, multi-task decoupling (separating human detection, object detection, and interaction prediction), and collaborative learning. Furthermore, accurately detecting rare or complex interactions remains a significant challenge. Current approaches, whether single-branch (all tasks in one decoder), two-branch (instance detection and interaction prediction), or three-branch (fully decoupled tasks), struggle to achieve an optimal balance. This paper seeks to overcome these limitations by proposing a unified framework that integrates the advantages of these paradigms while introducing a more effective learning strategy for complex interactions.
Methods
HOIMamba's core innovation lies in its Mamba-based decoder, designed to efficiently integrate and enhance HOI detection. The framework comprises:
- Global Feature Encoder: A standard CNN-Transformer architecture to extract visual features from input images.
- Mamba-based Decoder: This is the heart of HOIMamba. It's structured into three branches (human, object, interaction) that are designed to be efficient and disentangled.
- Cascaded Low-Rank Adaptations (LoRAs): Used within each branch to enable parameter-efficient fine-tuning, reducing computational cost while maintaining performance.
- Cross-enhance Mamba (CEM) Blocks: These blocks are designed for multi-view feature aggregation. They adaptively combine shallow and deep task-shared semantics, leveraging Mamba's selective state space model capabilities to enhance interaction features. CEM blocks facilitate cross-task interaction between human and object branches.
- Detection Context Propagation (DCP) Blocks: These blocks are crucial for information exchange. They transfer comprehensive interaction context from the interaction branch back to the human and object branches, enriching their representations and improving the detection of HOI instances.
- Progressive Learning Strategy: This strategy, powered by CEM and DCP blocks, aims to gradually extract interaction features from low to comprehensive levels. This is particularly beneficial for recognizing rare and complex interactions that require deeper semantic understanding.
Datasets
The paper evaluates HOIMamba on two standard HOI detection benchmarks:
- HICO-DET: A large-scale dataset for HOI detection.
- V-COCO: Another widely used dataset for HOI detection.
No new datasets were introduced in this paper.
Key Insights and Ideas
- Mamba as a Unified Decoder: The paper demonstrates that Mamba's architecture is well-suited for building a unified HOI decoder that can effectively balance efficiency, task decoupling, and collaborative learning, addressing limitations of previous Transformer-based decoders.
- Progressive Learning for Rare Interactions: The introduction of a progressive learning strategy (using CEM and DCP blocks) is key to improving performance on rare and complex HOI categories, which are often overlooked by methods that focus on general interaction patterns.
- Cross-Task Information Flow: The explicit design of DCP blocks to propagate interaction context back to human and object branches highlights the importance of bidirectional information flow for robust HOI detection.
- Efficiency Gains: By incorporating LoRAs and leveraging Mamba's inherent efficiency, HOIMamba achieves competitive or superior performance with fewer parameters and higher inference speeds compared to existing state-of-the-art methods.
Key Results
- State-of-the-Art Performance: HOIMamba achieves state-of-the-art results on both HICO-DET and V-COCO datasets. For instance, it achieves a full mAP of 41.51 (ResNet-50), 41.77 (ResNet-101), and 47.66 (Swin-Large), outperforming previous methods by significant margins (up to 15.6% relative improvement in mAP).
- Superiority in Rare HOIs: The model shows particularly strong improvements in detecting rare HOI categories, validating the effectiveness of the progressive learning strategy.
- Efficiency: HOIMamba is more efficient than comparable methods, with fewer parameters and higher FPS (27.26 FPS on a single RTX 3090) while achieving better performance. It shows a 36.0% relative improvement in FPS and a 15.6% reduction in FLOPS compared to QPIC.
Limitations and Future Work
- Visual Representation Limitations: The authors acknowledge a failure case where the model struggles to identify (human, pet, cat), attributing it to limitations in visual representation. This suggests that further improvements in feature extraction or context modeling might be needed for highly nuanced interactions.
- Future Directions: While not explicitly detailed as future work, the paper's success with Mamba suggests that exploring more advanced Mamba variants or integrating Mamba into other components of HOI detection pipelines could be promising avenues. The paper also implicitly points towards the need for better handling of visual ambiguities and fine-grained semantic understanding for challenging HOI scenarios.
Summary updated