TL;DR
This paper introduces SOV-STG-VLA, a novel framework for Human-Object Interaction (HOI) detection that disentangles the decoding process into subject, object, and verb prediction. By employing a Specific Target Guided (STG) denoising strategy and a Vision Language Advisor (VLA), the method achieves state-of-the-art performance with significantly reduced training time compared to existing approaches.
What Problem Does This Paper Address?
Human-Object Interaction (HOI) detection aims to identify not just objects in an image but also the relationships between humans and objects, specifically the action (verb) being performed. Existing transformer-based methods, while powerful, often suffer from long training times and complex optimization due to the intertwined nature of object detection and HOI recognition within a single decoding process. This entanglement leads to ambiguous representations for predicting both object/verb labels and bounding boxes. Furthermore, the gap between predicting HOI labels and verb labels, and effectively leveraging prior knowledge from Vision-Language Models (VLMs), remains a challenge.
Methods
The proposed SOV-STG-VLA framework tackles these issues with three core components:
- Subject-Object-Verb (SOV) Decoding: This approach splits the decoding process into three distinct parts: subject detection, object detection, and verb recognition. This disentanglement allows for more focused learning and reduces the complexity of each individual decoder. A novel "interaction region representation" is introduced for verb recognition.
- Specific Target Guided (STG) DeNoising: This training strategy guides the learning of label embeddings by using ground-truth information. It employs denoising queries to effectively transfer this information to both training and inference, accelerating convergence and improving accuracy. STG also leverages verb label embeddings to enhance verb recognition.
- Vision Language Advisor (VLA): This component integrates prior knowledge from pre-trained Vision-Language Models (VLMs) to further boost HOI recognition. It comprises two parts:
- Vision Advisor Decoder: This fuses global context information from the feature extractor and VLM with the spatial information of the interaction region.
- Verb-HOI (V-HOI) Bridge: This component bridges the gap between HOI label and verb label prediction, aligning the VLM's knowledge with the HOI recognition task.
Datasets
The paper evaluates its method on two widely used HOI detection datasets:
- HICO-DET: Contains 38,118 training images and 9,658 test images, with 117 verb classes and 80 object classes, forming 600 HOI classes.
- V-COCO: Contains 5,400 training images and 4,946 test images, with 80 object classes and 29 verb classes.
Key Insights and Ideas
- Disentanglement is Key: Splitting the HOI decoding into subject, object, and verb prediction significantly simplifies the learning process and leads to faster convergence.
- Guided Training Accelerates Learning: The STG denoising strategy, by explicitly using ground-truth information to guide label embedding learning, dramatically speeds up training and improves performance.
- Leveraging VLMs Effectively: The VLA component demonstrates a sophisticated way to integrate VLM knowledge, not just by using pre-trained models, but by carefully aligning them with the specific HOI detection task and interaction region.
- Verb Box as a Novel Representation: Introducing a specific verb box representation (ASMBR) helps constrain the verb feature extraction and focus on the relevant interaction region.
Key Results
The SOV-STG-VLA framework achieves state-of-the-art (SOTA) performance on the HICO-DET dataset, notably improving accuracy while requiring only one-sixth of the training epochs compared to previous SOTA methods. For instance, SOV-STG-VLA-S achieves 41.16 mAP on HICO-DET's Full category (Default setting) in just 15 epochs. On V-COCO, it also shows significant improvements, outperforming SOV-STG-S by over 4% in both scenarios. Ablation studies confirm the effectiveness of each proposed module, with the STG strategy and the vision advisor decoder showing substantial contributions.
Limitations and Future Work
The paper does not explicitly detail limitations in a dedicated section. However, the authors suggest future work will involve exploring the potential of the proposed framework in other vision-language tasks and further improving the model's performance. The reliance on pre-trained VLMs and object detectors (like DAB-DETR) implies that the performance is also dependent on the quality of these upstream components.
Summary updated