Introduction
Human-Object Interaction (HOI) detection is a critical computer vision task that involves identifying triplets of <human, verb, object> within an image. Unlike standard object detection, HOI requires understanding the semantic relationship between entities—for instance, distinguishing whether a person is holding a phone or repairing it.
Modern HOI detection frameworks frequently leverage the DEtection TRansformer (DETR) architecture. In these models, "queries"—learnable vectors—are used to probe image features for relevant information. Traditionally, these queries are initialized with random Gaussian noise. However, this "vague" starting point often leads to slow convergence and sub-optimal performance, as the model must learn to localize and classify complex interactions from scratch.
Methodology: Dual Query Enhancement
The Dual Query Enhancement Network (DQEN) addresses the limitations of random initialization by providing the transformer decoders with high-quality "coarse guesses" or priors. The architecture consists of two primary enhancement modules: Object Query Enhancement (OQE) and Interaction Query Enhancement (IQE).
Object Query Enhancement (OQE)
Instead of starting with noise, OQE utilizes encoder features passed through an initial object classifier. The model selects the "Top-N" features with the highest confidence scores to initialize the object queries. This ensures that the decoder begins its search in regions likely to contain relevant objects. To prevent the model from over-fitting to simple object detection at the expense of interaction understanding, the authors employ gradient truncation during this process.
Interaction Query Enhancement (IQE) and Semantic Fusion
The IQE module leverages the zero-shot capabilities of the CLIP (Contrastive Language-Image Pre-training) model. By calculating the similarity between the input image and a set of HOI text labels (e.g., "a photo of a person riding a horse"), the module identifies the most likely interaction candidates. These candidates are then processed by the Interaction Semantic Fusion (ISF) unit, which decouples the "verb" and "object" components and fuses them into a semantic feature vector ($Q_i$) used to initialize interaction queries.
HOI Decoder and Auxiliary Prediction
The enhanced queries are fed into a two-stage decoder. The Instance Decoder refines human and object features ($V_h, V_o$), while the Interaction Decoder processes the fused interaction queries ($Q_{inter}$). An Auxiliary Prediction Unit (APU), consisting of an additional verb classifier, further sharpens the model's understanding by reusing semantic features to refine the final interaction scores.
Key Insights
- Human-Centric Priors: Enhancing object queries surprisingly improves the model's focus on the humans involved in interactions. By localizing the object first, the model effectively constrains the search space for the associated human actor.
- CLIP as a Coarse Guesser: Even when CLIP is not perfectly accurate, its top-8 matches cover over 50% of ground-truth interactions. This provides a significantly better starting point than random noise, leading to faster and more stable training.
- Zero-Shot Generalization: By incorporating semantic knowledge from CLIP, DQEN demonstrates superior performance in "unseen verb" scenarios, outperforming previous state-of-the-art models in zero-shot benchmarks.
Experimental Results
DQEN was evaluated on the HICO-Det and V-COCO benchmarks. On HICO-Det, DQEN achieved 35.23 mAP, a 1.48 point improvement over the GEN-VLKT baseline. Notably, the performance gain on "Rare" interactions was even more pronounced, reaching +3.54 mAP. In zero-shot settings, DQEN outperformed HOICLIP by 1.15 mAP. Despite the additional modules, the network maintains a competitive inference speed of 29.02 FPS, suitable for near-real-time applications.
Conclusion and Future Work
DQEN demonstrates that informed query initialization is a powerful tool for improving DETR-based HOI detection. While the model shows significant gains on large-scale datasets like HICO-Det, improvements on V-COCO are more marginal due to the dataset's high density of overlapping human-object pairs. Future research will focus on enhancing query robustness in these crowded environments where multiple interactions occur in close proximity.
Summary updated