Introduction
Human-Object Interaction (HOI) detection is a critical task for understanding visual scenes. However, traditional methods are often constrained to a small set of object categories (e.g., the 80 classes in COCO). Scaling to thousands of categories introduces significant challenges, including detector failure on rare objects and a combinatorial explosion of potential person-object pairs. This paper proposes a novel human-centric query approach that treats the human as the primary anchor for searching interacting objects across multiple scales.
Methodology
The proposed framework shifts the paradigm from bottom-up object detection to a top-down, human-centric search. It consists of three core stages:
- Person Query Generation: Using a standard backbone, the model first identifies humans in the scene. Each person is represented by a query vector encoding visual and spatial information.
- Interaction-as-Query: This vector acts as a query in a Transformer-based decoder. Instead of generic object detection, the model generates attention maps specifically tuned to find objects the person is likely interacting with.
- Cascaded Multi-Scale Detection: To accurately detect small or rare objects, the model employs a "zoom-in" strategy. It starts with coarse feature maps (P5) and iteratively refines the search on higher-resolution maps (P4 to P2) for regions of interest.
Datasets
To evaluate the model in large-vocabulary settings, the authors introduced two benchmarks:
- SWiG-HOI: Derived from the Semantic Web Image Graph, it contains 1,000 object categories and 406 actions across ~59k images.
- DOH-HOI: A subset of the "100 Days of Hands" dataset, annotated with 300 object categories specifically for hand-object interactions.
Results and Key Insights
The "top-down" approach proves highly effective for long-tail distributions. On SWiG-HOI, the method achieved a ~20 point improvement in Average Recall (AR) for the top 10 boxes compared to standard detectors. The cascaded detection mechanism was crucial for small objects, which often get "washed out" in global attention maps. Furthermore, the model is highly efficient, achieving an inference speed of 93ms per image, significantly faster than previous one-stage HOI detectors.
Conclusion
By coupling action prediction with object detection through a human-centric query, this work addresses the scalability issues of traditional HOI models. While challenges remain in zero-shot generalization to unseen verb-object pairs, the cascaded multi-scale approach provides a robust foundation for large-vocabulary scene understanding.
Summary updated