Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration

Juhan Park, Kyungjae Lee, Hyung Jin Chang, Jungchan ChoarXiv 2025 · DOI · arXiv

Abstract

Human-Object Interaction (HOI) detection is a critical task for understanding complex scenes, yet traditional methods are limited to bounding-box triplets. This paper presents Seg2HOI, a framework that upgrades HOI detection to pixel-level quadruplets (<human, object, action, union-mask>). By employing a lightweight, implicit relation decoder atop a frozen vision foundation model and utilizing pseudo-labeled masks, Seg2HOI achieves state-of-the-art performance. The framework also enables interactive, prompt-driven HOI segmentation, facilitating applications in robotics and augmented reality without requiring extensive retraining.

Introduction

Standard HOI models typically output a triplet consisting of a human bounding box, an object bounding box, and an action label. While useful, this representation is insufficient for high-precision tasks like collaborative robotics or advanced AR, where a pixel-level understanding of the interaction is necessary. Transitioning to segmentation masks faces two primary obstacles: the lack of pixel-level HOI annotations in existing datasets and the high computational cost of fine-tuning massive vision foundation models. Seg2HOI overcomes these by leveraging frozen foundation models and a novel pseudo-labeling strategy.

HOI segmentation output showing pixel-level quadruplets for the 'ride horse' interaction.

Methodology

Seg2HOI integrates a specialized HOI decoder with a pretrained, frozen segmentation foundation model such as Mask-DINO or SEEM.

Frozen Foundation Model and Implicit Learning

The base model remains frozen to preserve its robust segmentation and detection capabilities. On top of this, Seg2HOI introduces an implicit human-object relation learning mechanism. Instead of evaluating all possible pairs, the model uses cross-attention blocks where human and object queries learn to identify interacting counterparts efficiently.

Mask Pseudo-Labeling

To address the absence of ground-truth masks in datasets like HICO-DET, the authors generate pseudo-labels by matching foundation model instance masks to ground-truth bounding boxes. These are combined to form union and intersection masks, providing the necessary supervision for training the HOI decoder.

Results and Discussion

The inclusion of mask supervision significantly enhances performance across standard benchmarks.

Performance Benchmarks

On the V-COCO dataset, Seg2HOI achieved a role AP of 70.7 (Scenario 1) and 73.0 (Scenario 2) with a Swin-L backbone. On HICO-DET, it reached 37.98 mAP, outperforming models that did not utilize backbone fine-tuning.

Zero-Shot and Interactive Capabilities

A key finding is that mask-guided learning improves zero-shot performance on unseen objects by 89.9%. Furthermore, by integrating with vision-language models like SEEM, Seg2HOI supports interactive prompts (text or visual), allowing for flexible, open-vocabulary interaction segmentation.

Interactive robotics application demonstrating prompt-driven HOI segmentation for complex tasks.

Conclusion

Seg2HOI demonstrates that high-level relational semantics can be learned through a lightweight decoder without disturbing the underlying foundation model. While challenges such as intersection mask noise and accuracy on completely unseen objects persist, the framework's scalability suggests a promising path toward comprehensive Object-Object Interaction (OOI) modeling.

Summary updated

Abstract

In this work, we introduce Segmentation to Human-Object Interaction (\textit{\textbf{Seg2HOI}}) approach, a novel framework that integrates segmentation-based vision foundation models with the human-object interaction task, distinguished from traditional detection-based Human-Object Interaction (HOI) methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human-object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates a decoder that applies these attributes to HOI task. Despite training only for HOI, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.
BibTeX
@misc{park2025foundation,
  author = {Juhan Park and Kyungjae Lee and Hyung Jin Chang and Jungchan Cho},
  title = {{Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration}},
  year = {2025},
  eprint = {2504.19847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.19847},
}
Figure