Open-Category Human-Object Interaction Pre-training via Language Modeling Framework

Sipeng Zheng, Boshen Xu, Qin JinCVPR 2023 · DOI

Introduction

Identifying how humans interact with objects—such as distinguishing between a person "riding" a horse versus "feeding" it—is a cornerstone of computer vision. Traditional Human-Object Interaction (HOI) detection models have historically operated within a "closed-set" paradigm, limited to recognizing a fixed vocabulary of actions (e.g., the 117 relations in the HICO-DET dataset). This approach fails when encountering novel interactions and suffers from a severe "long-tail" problem, where common actions are over-represented while rare interactions lack sufficient labeled data.

Conceptual overview of OpenCat's approach to handling the long-tail and unseen HOIs compared to traditional fixed-logit classifiers.

To overcome these limitations, we present OpenCat, a framework that reformulates HOI detection as a generative language modeling task. By treating HOI triplets as sequences of tokens, OpenCat leverages massive amounts of weakly supervised image-caption data to achieve state-of-the-art performance on both common and unseen interactions.

Methodology: HOI as Sequence Generation

OpenCat abandons the traditional discriminative approach of outputting fixed-dimension logits. Instead, it treats HOI as a sequence generation task. An HOI instance is serialized into a string of tokens: [pointer_h, pointer_o, relation, object]. The "pointers" are discrete tokens representing indices of bounding boxes identified by an offline object detector.

Architecture and Proxy Tasks

The model utilizes a standard visual-language encoder-decoder structure. To ground the model's understanding of interactions without requiring expensive manual labels, we employ four specialized pre-training proxy tasks on a dataset of 750,000 image-caption pairs:

  1. Masked Language Prediction (MLP): The model must fill in missing parts of an HOI triplet based on visual context.
  2. Human-Object Relation Prediction (HRP): The model generates all possible HOI triplets for an image from a generic prompt.
  3. Human-Object Patch Jigsaw (HPJ): By shuffling and rotating patches of human-object regions, the model is forced to learn spatial relationships and geometric consistency.
  4. Soft Relational Matching (SRM): Using pre-trained Vision-Language models (like CLIP) as teachers, the model generates "soft" labels that bridge the gap between raw captions and specific image regions.
The OpenCat architecture featuring a visual-language encoder-decoder and the four specialized pre-training proxy tasks: MLP, HRP, HPJ, and SRM.

Experimental Results

OpenCat sets new benchmarks across major HOI datasets, including HICO-DET, V-COCO, and MPII.

  • HICO-DET Performance: The model achieved 32.68 mAP on the Full set, significantly outperforming previous state-of-the-art methods like CDN.
  • Addressing the Long-Tail: On the "Rare" subset of HICO-DET, OpenCat reached 28.42 mAP, demonstrating superior handling of infrequent interactions.
  • Zero-Shot Capabilities: In "Unseen Combination" scenarios—where the model must detect interactions never encountered during supervised training—OpenCat outperformed competitors by over 11% mAP.

Qualitative analysis confirms that OpenCat successfully identifies rare interactions, such as "washing a dog" or "jumping a bicycle," where baseline models often default to generic, incorrect verbs like "holding."

Discussion and Limitations

While OpenCat represents a significant leap forward, challenges remain. The auto-regressive nature of the decoder results in slower inference speeds compared to one-shot classification models. Additionally, images with a high density of interactions result in long sequences that can challenge the attention mechanism. Future work will focus on integrating the object detector into an end-to-end trainable sequence generator to further streamline the pipeline.

Summary updated

BibTeX
@inproceedings{zheng2023open,
  author = {Sipeng Zheng and Boshen Xu and Qin Jin},
  title = {{Open-Category Human-Object Interaction Pre-training via Language Modeling Framework}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01858},
}
Figure