Introduction
Identifying how humans interact with objects—such as distinguishing between a person "riding" a horse versus "feeding" it—is a cornerstone of computer vision. Traditional Human-Object Interaction (HOI) detection models have historically operated within a "closed-set" paradigm, limited to recognizing a fixed vocabulary of actions (e.g., the 117 relations in the HICO-DET dataset). This approach fails when encountering novel interactions and suffers from a severe "long-tail" problem, where common actions are over-represented while rare interactions lack sufficient labeled data.
To overcome these limitations, we present OpenCat, a framework that reformulates HOI detection as a generative language modeling task. By treating HOI triplets as sequences of tokens, OpenCat leverages massive amounts of weakly supervised image-caption data to achieve state-of-the-art performance on both common and unseen interactions.
Methodology: HOI as Sequence Generation
OpenCat abandons the traditional discriminative approach of outputting fixed-dimension logits. Instead, it treats HOI as a sequence generation task. An HOI instance is serialized into a string of tokens: [pointer_h, pointer_o, relation, object]. The "pointers" are discrete tokens representing indices of bounding boxes identified by an offline object detector.
Architecture and Proxy Tasks
The model utilizes a standard visual-language encoder-decoder structure. To ground the model's understanding of interactions without requiring expensive manual labels, we employ four specialized pre-training proxy tasks on a dataset of 750,000 image-caption pairs:
- Masked Language Prediction (MLP): The model must fill in missing parts of an HOI triplet based on visual context.
- Human-Object Relation Prediction (HRP): The model generates all possible HOI triplets for an image from a generic prompt.
- Human-Object Patch Jigsaw (HPJ): By shuffling and rotating patches of human-object regions, the model is forced to learn spatial relationships and geometric consistency.
- Soft Relational Matching (SRM): Using pre-trained Vision-Language models (like CLIP) as teachers, the model generates "soft" labels that bridge the gap between raw captions and specific image regions.
Experimental Results
OpenCat sets new benchmarks across major HOI datasets, including HICO-DET, V-COCO, and MPII.
- HICO-DET Performance: The model achieved 32.68 mAP on the Full set, significantly outperforming previous state-of-the-art methods like CDN.
- Addressing the Long-Tail: On the "Rare" subset of HICO-DET, OpenCat reached 28.42 mAP, demonstrating superior handling of infrequent interactions.
- Zero-Shot Capabilities: In "Unseen Combination" scenarios—where the model must detect interactions never encountered during supervised training—OpenCat outperformed competitors by over 11% mAP.
Qualitative analysis confirms that OpenCat successfully identifies rare interactions, such as "washing a dog" or "jumping a bicycle," where baseline models often default to generic, incorrect verbs like "holding."
Discussion and Limitations
While OpenCat represents a significant leap forward, challenges remain. The auto-regressive nature of the decoder results in slower inference speeds compared to one-shot classification models. Additionally, images with a high density of interactions result in long sequences that can challenge the attention mechanism. Future work will focus on integrating the object detector into an end-to-end trainable sequence generator to further streamline the pipeline.
Summary updated