Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection

Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wu, Junsong Yuan, Yap-Peng Tan

Introduction

Human-Object Interaction (HOI) detection is a critical task for understanding visual scenes. However, traditional methods are often constrained to a small set of object categories (e.g., the 80 classes in COCO). Scaling to thousands of categories introduces significant challenges, including detector failure on rare objects and a combinatorial explosion of potential person-object pairs. This paper proposes a novel human-centric query approach that treats the human as the primary anchor for searching interacting objects across multiple scales.

Examples of large-vocabulary HOI detection and the long-tail distribution of object and interaction categories.

Methodology

The proposed framework shifts the paradigm from bottom-up object detection to a top-down, human-centric search. It consists of three core stages:

  1. Person Query Generation: Using a standard backbone, the model first identifies humans in the scene. Each person is represented by a query vector encoding visual and spatial information.
  2. Interaction-as-Query: This vector acts as a query in a Transformer-based decoder. Instead of generic object detection, the model generates attention maps specifically tuned to find objects the person is likely interacting with.
  3. Cascaded Multi-Scale Detection: To accurately detect small or rare objects, the model employs a "zoom-in" strategy. It starts with coarse feature maps (P5) and iteratively refines the search on higher-resolution maps (P4 to P2) for regions of interest.
The unified architecture featuring person query generation and the cascaded multi-scale detection pipeline.

Datasets

To evaluate the model in large-vocabulary settings, the authors introduced two benchmarks:

  • SWiG-HOI: Derived from the Semantic Web Image Graph, it contains 1,000 object categories and 406 actions across ~59k images.
  • DOH-HOI: A subset of the "100 Days of Hands" dataset, annotated with 300 object categories specifically for hand-object interactions.

Results and Key Insights

The "top-down" approach proves highly effective for long-tail distributions. On SWiG-HOI, the method achieved a ~20 point improvement in Average Recall (AR) for the top 10 boxes compared to standard detectors. The cascaded detection mechanism was crucial for small objects, which often get "washed out" in global attention maps. Furthermore, the model is highly efficient, achieving an inference speed of 93ms per image, significantly faster than previous one-stage HOI detectors.

Conclusion

By coupling action prediction with object detection through a human-centric query, this work addresses the scalability issues of traditional HOI models. While challenges remain in zero-shot generalization to unseen verb-object pairs, the cascaded multi-scale approach provides a robust foundation for large-vocabulary scene understanding.

Summary updated

Abstract

In this work, we study the problem of human-object interaction (HOI) detection with large vocabulary object categories. Previous HOI studies are mainly conducted in the regime of limit object categories (e.g., 80 categories). Their solutions may face new difficulties in both object detection and interaction classification due to the increasing diversity of objects (e.g., 1000 categories). Different from previous methods, we formulate the HOI detection as a query problem. We propose a unified model to jointly discover the target objects and predict the corresponding interactions based on the human queries, thereby eliminating the need of using generic object detectors, extra steps to associate human-object instances, and multi-stream interaction recognition. This is achieved by a repurposed Transformer unit and a novel cascade detection over multi-scale feature maps. We observe that such a highly-coupled solution brings benefits for both object detection and interaction classification in a large vocabulary setting. To study the new challenges of the large vocabulary HOI detection, we assemble two datasets from the publicly available SWiG and 100 Days of Hands datasets. Experiments on these datasets validate that our proposed method can achieve a notable mAP improvement on HOI detection with a faster inference speed than existing one-stage HOI detectors. Our code is available at https://github.com/scwangdyd/large_vocabulary_hoi_detection.
BibTeX
@inproceedings{wang2021discovering,
  author = {Suchen Wang and Kim-Hui Yap and Henghui Ding and Jiyan Wu and Junsong Yuan and Yap-Peng Tan},
  title = {{Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01322},
}
Figure