Abstract
Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Recently, Contrastive Language-Image Pre-training (CLIP) has shown great potential in providing interaction prior for HOI detectors via knowledge distillation. However, such approaches often rely on large-scale training data and suffer from inferior performance under few/zero-shot scenarios. In this paper, we propose a novel HOI detection framework that efficiently extracts prior knowledge from CLIP and achieves better generalization. In detail, we first introduce a novel interaction decoder to extract informative regions in the visual feature map of CLIP via a cross-attention mechanism, which is then fused with the detection backbone by a knowledge integration block for more accurate human-object pair detection. In addition, prior knowledge in CLIP text encoder is leveraged to generate a classifier by embedding HOI descriptions. To distinguish fine-grained interactions, we build a verb classifier from training data via visual semantic arithmetic and a lightweight verb representation adapter. Furthermore, we propose a training-free enhancement to exploit global HOI predictions from CLIP. Extensive experiments demonstrate that our method outperforms the state of the art by a large margin on various settings, e.g. +4.04 mAP on HICO-Det.
Introduction
Human-Object Interaction (HOI) detection is the task of identifying "who is doing what to what" in an image (e.g., a person riding a bicycle). This is notoriously difficult because interactions are compositional—there are thousands of possible combinations of humans, objects, and verbs—and real-world data is "long-tailed," meaning most interactions are rarely seen in training sets.
Recently, researchers have tried to use CLIP (a massive Vision-Language Model) to help HOI detectors. However, most previous attempts used knowledge distillation, where a "student" detector tries to mimic a "teacher" CLIP model. This is inefficient: it requires massive amounts of data to work well and often fails to generalize to "zero-shot" interactions (combinations never seen during training). HOICLIP addresses this by moving away from distillation toward efficient knowledge retrieval, achieving state-of-the-art results, particularly in zero-shot and low-data scenarios.
Methodology
HOICLIP focuses on retrieving knowledge directly from the pre-trained CLIP model. The architecture consists of three main innovations:
- Interaction Decoder with Knowledge Integration: Instead of just using a standard detection backbone, the model introduces a decoder that uses cross-attention to look at both the detector's own features and CLIP’s spatial feature maps. This allows the model to "query" CLIP for specific regional information relevant to an interaction.
- Visual Semantic Arithmetic (VSA): To better represent verbs, the authors use a "visual arithmetic" logic. They hypothesize that a verb's visual representation can be calculated as the difference between the features of the entire HOI region and the features of just the object ($Verb \approx HOI - Object$). This helps the model isolate the "action" from the "entities."
- Zero-shot HOI Enhancement: This training-free branch uses CLIP’s text encoder. By feeding in prompts like "A photo of a person [verb-ing] a [object]," the model generates a classifier that can recognize unseen interactions during inference without needing specific training examples.
Experiments and Results
The authors evaluated HOICLIP on two primary benchmarks: HICO-DET (600 HOI categories) and V-COCO. They also tested several challenging zero-shot settings, including Rare First Unseen Combination (RF-UC), Unseen Verb (UV), and Unseen Object (UO).
Key Results
- Zero-Shot Dominance: On the HICO-DET "Non-rare First Unseen" setting, HOICLIP achieved a +4.04 mAP gain over the previous state-of-the-art (GEN-VLKT).
- Data Efficiency: When trained on only 25% of the available data, HOICLIP’s performance on rare categories was 81.41% higher than its closest competitor. This suggests the model is far more robust when data is scarce.
- Competitive Performance: On the standard HICO-DET benchmark (fully supervised), it reached 34.69 mAP, setting a new high bar for ResNet-50-based models.
Discussion and Conclusion
The success of HOICLIP highlights that retrieval is often superior to distillation for large pre-trained models. It is more effective to build a model that knows how to ask CLIP for the right information at the right time. Furthermore, HOICLIP proves that CLIP’s internal spatial feature maps contain rich, localized information about object relationships, and the $HOI - Object = Verb$ formula provides a simple yet powerful way to handle the long-tail problem.
While HOICLIP is slightly slower due to the extra CLIP forward pass (approx. 55ms vs 52ms), its gains in generalization and data efficiency are significant. Future work may involve fine-tuning the CLIP visual encoder efficiently or reducing the computational cost of the transformer-based architecture for real-time applications.
Summary updated