Abstract
Human-Object Interaction (HOI) detection is a critical task for understanding complex scenes, yet traditional methods are limited to bounding-box triplets. This paper presents Seg2HOI, a framework that upgrades HOI detection to pixel-level quadruplets (<human, object, action, union-mask>). By employing a lightweight, implicit relation decoder atop a frozen vision foundation model and utilizing pseudo-labeled masks, Seg2HOI achieves state-of-the-art performance. The framework also enables interactive, prompt-driven HOI segmentation, facilitating applications in robotics and augmented reality without requiring extensive retraining.
Introduction
Standard HOI models typically output a triplet consisting of a human bounding box, an object bounding box, and an action label. While useful, this representation is insufficient for high-precision tasks like collaborative robotics or advanced AR, where a pixel-level understanding of the interaction is necessary. Transitioning to segmentation masks faces two primary obstacles: the lack of pixel-level HOI annotations in existing datasets and the high computational cost of fine-tuning massive vision foundation models. Seg2HOI overcomes these by leveraging frozen foundation models and a novel pseudo-labeling strategy.
Methodology
Seg2HOI integrates a specialized HOI decoder with a pretrained, frozen segmentation foundation model such as Mask-DINO or SEEM.
Frozen Foundation Model and Implicit Learning
The base model remains frozen to preserve its robust segmentation and detection capabilities. On top of this, Seg2HOI introduces an implicit human-object relation learning mechanism. Instead of evaluating all possible pairs, the model uses cross-attention blocks where human and object queries learn to identify interacting counterparts efficiently.
Mask Pseudo-Labeling
To address the absence of ground-truth masks in datasets like HICO-DET, the authors generate pseudo-labels by matching foundation model instance masks to ground-truth bounding boxes. These are combined to form union and intersection masks, providing the necessary supervision for training the HOI decoder.
Results and Discussion
The inclusion of mask supervision significantly enhances performance across standard benchmarks.
Performance Benchmarks
On the V-COCO dataset, Seg2HOI achieved a role AP of 70.7 (Scenario 1) and 73.0 (Scenario 2) with a Swin-L backbone. On HICO-DET, it reached 37.98 mAP, outperforming models that did not utilize backbone fine-tuning.
Zero-Shot and Interactive Capabilities
A key finding is that mask-guided learning improves zero-shot performance on unseen objects by 89.9%. Furthermore, by integrating with vision-language models like SEEM, Seg2HOI supports interactive prompts (text or visual), allowing for flexible, open-vocabulary interaction segmentation.
Conclusion
Seg2HOI demonstrates that high-level relational semantics can be learned through a lightweight decoder without disturbing the underlying foundation model. While challenges such as intersection mask noise and accuracy on completely unseen objects persist, the framework's scalability suggests a promising path toward comprehensive Object-Object Interaction (OOI) modeling.
Summary updated