Abstract
Human-Object Interaction (HOI) detection is a critical task in computer vision that involves identifying relationships between humans and objects in a scene. Traditional methods treat HOI as a classification problem over a fixed set of labels, which limits their ability to generalize to unseen or rare interactions. We introduce THID (Transferable Human-Object Interaction Detector), a one-stage detector that utilizes natural language supervision to achieve zero-shot transferability. By aligning visual features with semantic text embeddings, THID can recognize complex interactions it has never encountered during training. Our approach significantly outperforms previous zero-shot methods on the HICO-DET and SWIG-HOI benchmarks.
Introduction: The Challenge of Combinatorial Explosion
The core difficulty in HOI detection is the "combinatorial explosion" of possible interactions. With hundreds of actions and thousands of objects, the number of potential combinations is vast. Collecting and labeling data for every possible pair—such as "washing a tiger" or "riding a vacuum cleaner"—is practically impossible.
Traditional detectors are "brittle" because they rely on a closed set of discrete labels. If a model hasn't seen a specific combination during training, it cannot categorize it. THID addresses this by moving away from fixed labels toward a transferable framework that uses the semantic relationships inherent in language to generalize to novel interactions.
Methodology
THID is a one-stage detector that bridges computer vision and natural language processing through two primary branches: a visual encoder and a text encoder.
HOI Visual Encoder
The visual branch is built upon a Vision Transformer (ViT). Unlike standard ViT architectures that use a single [CLS] token for global image representation, THID introduces multiple specialized [HOI] tokens. These tokens are designed to "seek out" and localize different interactions within the scene. To prevent redundant detections, the authors implemented an HOI Sequence Parser. This parser uses a causal mask, ensuring that each [HOI] token is aware of the detections made by its predecessors, effectively performing a "soft" internal non-maximum suppression.
Text Encoding and Prompt Engineering
Rather than using static templates (e.g., "a photo of a person [action] [object]"), THID employs learnable context tokens ([PREFIX] and [CONJUN]). This allows the model to learn the optimal linguistic structure for representing interactions. Furthermore, providing detailed definitions of actions—such as defining "washing" as "removing a stain or dirt"—provides the text encoder with more semantic "hooks" to align with visual features.
Knowledge Distillation from CLIP
THID leverages the pre-trained CLIP (Contrastive Language-Image Pre-training) model. By mapping visual features and text descriptions into a shared latent space, the model can recognize an unseen interaction (like "riding a horse") because the visual features align with the linguistic representation of that phrase.
Key Insights
- Language as a Semantic Bridge: By framing HOI as a visual-to-text matching problem, the model gains a form of "common sense." It can infer the appearance of novel combinations based on its understanding of individual components (actions and objects).
- Sequential vs. Parallel Processing: The Sequence Parser is a critical architectural innovation. While many transformer-based detectors process queries independently (in parallel), forcing them into a sequence encourages unique and diverse detections.
- Semantic Depth: Detailed action definitions significantly boost performance, highlighting that the quality of linguistic supervision is as important as the visual data.
Results and Evaluation
The model was evaluated on two major benchmarks: HICO-DET (600 categories) and SWIG-HOI (400 actions, 1,000 objects).
- Zero-Shot Performance: On unseen interactions in HICO-DET, THID achieved 17.53 mAP, a substantial improvement over the previous state-of-the-art of 13.16 mAP.
- Scalability: On the more challenging SWIG-HOI dataset, THID reached 10.04 mAP for unseen interactions, nearly doubling the performance of baseline one-stage detectors.
- Generalization: THID maintains competitive performance on "seen" categories while excelling at "unseen" ones, demonstrating a balanced ability to generalize without losing accuracy on known data.
Conclusion and Future Work
THID represents a significant step toward open-vocabulary human-object interaction detection. By utilizing natural language as a supervisory signal, it overcomes the limitations of fixed-label systems. However, limitations remain, including a fixed input resolution (224x224) that can hinder the detection of small objects and a higher computational cost compared to CNN-based models. Future research will focus on integrating multi-scale features to improve localization and optimizing the distillation process for real-time applications.
Summary updated