Learning Transferable Human-Object Interaction Detector with Natural Language Supervision

Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, Junsong Yuan

Abstract

Human-Object Interaction (HOI) detection is a critical task in computer vision that involves identifying relationships between humans and objects in a scene. Traditional methods treat HOI as a classification problem over a fixed set of labels, which limits their ability to generalize to unseen or rare interactions. We introduce THID (Transferable Human-Object Interaction Detector), a one-stage detector that utilizes natural language supervision to achieve zero-shot transferability. By aligning visual features with semantic text embeddings, THID can recognize complex interactions it has never encountered during training. Our approach significantly outperforms previous zero-shot methods on the HICO-DET and SWIG-HOI benchmarks.

Introduction: The Challenge of Combinatorial Explosion

The core difficulty in HOI detection is the "combinatorial explosion" of possible interactions. With hundreds of actions and thousands of objects, the number of potential combinations is vast. Collecting and labeling data for every possible pair—such as "washing a tiger" or "riding a vacuum cleaner"—is practically impossible.

Comparison between traditional HOI detectors and the proposed joint visual-and-text modeling approach.

Traditional detectors are "brittle" because they rely on a closed set of discrete labels. If a model hasn't seen a specific combination during training, it cannot categorize it. THID addresses this by moving away from fixed labels toward a transferable framework that uses the semantic relationships inherent in language to generalize to novel interactions.

Methodology

THID is a one-stage detector that bridges computer vision and natural language processing through two primary branches: a visual encoder and a text encoder.

The architecture of THID, featuring the HOI Visual Encoder with a sequence parser and the Text Encoder with learnable prompts.

HOI Visual Encoder

The visual branch is built upon a Vision Transformer (ViT). Unlike standard ViT architectures that use a single [CLS] token for global image representation, THID introduces multiple specialized [HOI] tokens. These tokens are designed to "seek out" and localize different interactions within the scene. To prevent redundant detections, the authors implemented an HOI Sequence Parser. This parser uses a causal mask, ensuring that each [HOI] token is aware of the detections made by its predecessors, effectively performing a "soft" internal non-maximum suppression.

Text Encoding and Prompt Engineering

Rather than using static templates (e.g., "a photo of a person [action] [object]"), THID employs learnable context tokens ([PREFIX] and [CONJUN]). This allows the model to learn the optimal linguistic structure for representing interactions. Furthermore, providing detailed definitions of actions—such as defining "washing" as "removing a stain or dirt"—provides the text encoder with more semantic "hooks" to align with visual features.

Knowledge Distillation from CLIP

THID leverages the pre-trained CLIP (Contrastive Language-Image Pre-training) model. By mapping visual features and text descriptions into a shared latent space, the model can recognize an unseen interaction (like "riding a horse") because the visual features align with the linguistic representation of that phrase.

Key Insights

  • Language as a Semantic Bridge: By framing HOI as a visual-to-text matching problem, the model gains a form of "common sense." It can infer the appearance of novel combinations based on its understanding of individual components (actions and objects).
  • Sequential vs. Parallel Processing: The Sequence Parser is a critical architectural innovation. While many transformer-based detectors process queries independently (in parallel), forcing them into a sequence encourages unique and diverse detections.
  • Semantic Depth: Detailed action definitions significantly boost performance, highlighting that the quality of linguistic supervision is as important as the visual data.

Results and Evaluation

The model was evaluated on two major benchmarks: HICO-DET (600 categories) and SWIG-HOI (400 actions, 1,000 objects).

  • Zero-Shot Performance: On unseen interactions in HICO-DET, THID achieved 17.53 mAP, a substantial improvement over the previous state-of-the-art of 13.16 mAP.
  • Scalability: On the more challenging SWIG-HOI dataset, THID reached 10.04 mAP for unseen interactions, nearly doubling the performance of baseline one-stage detectors.
  • Generalization: THID maintains competitive performance on "seen" categories while excelling at "unseen" ones, demonstrating a balanced ability to generalize without losing accuracy on known data.

Conclusion and Future Work

THID represents a significant step toward open-vocabulary human-object interaction detection. By utilizing natural language as a supervisory signal, it overcomes the limitations of fixed-label systems. However, limitations remain, including a fixed input resolution (224x224) that can hinder the detection of small objects and a higher computational cost compared to CNN-based models. Future research will focus on integrating multi-scale features to improve localization and optimizing the distillation process for real-time applications.

Summary updated

Abstract

It is difficult to construct a data collection including all possible combinations of human actions and interacting objects due to the combinatorial nature of human-object interactions (HOI). In this work, we aim to develop a transferable HOI detector for unseen interactions. Existing HOI detectors often treat interactions as discrete labels and learn a classifier according to a predetermined category space. This is inherently inapt for detecting unseen interactions which are out of the predefined categories. Conversely, we treat independent HOI labels as the natural language supervision of interactions and embed them into a joint visual-and-text space to capture their correlations. More specifically, we propose a new HOI visual encoder to detect the interacting humans and objects, and map them to a joint feature space to perform interaction recognition. Our visual encoder is instantiated as a Vision Transformer with new learnable HOI tokens and a sequence parser to generate unique HOI predictions. It distills and leverages the transferable knowledge from the pretrained CLIP model to perform the zero-shot interaction detection. Experiments on two datasets, SWIG-HOI and HICO-DET, validate that our proposed method can achieve a notable mAP improvement on detecting both seen and unseen HOIs. Our code is available at https://github.com/scwangdyd/promting_hoi.
BibTeX
@inproceedings{wang2022learning,
  author = {Suchen Wang and Yueqi Duan and Henghui Ding and Yap-Peng Tan and Kim-Hui Yap and Junsong Yuan},
  title = {{Learning Transferable Human-Object Interaction Detector with Natural Language Supervision}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00101},
}
Figure