QPIC: Query-Based Pairwise Human-Object Interaction Detection With Image-Wide Contextual Information

Masato Tamura, Hiroki Ohashi, Tomoaki YoshinagaCVPR 2021 · DOI

Abstract

Human-Object Interaction (HOI) detection is a fundamental computer vision task that requires identifying human-object pairs and classifying their interactions. Traditional approaches often struggle with spatial complexity, either losing context through localized cropping or failing in crowded scenes due to heuristic-based feature extraction. We propose QPIC (Query-Based Pairwise Interaction Classification), a Transformer-based architecture that treats HOI detection as a set-prediction problem. By leveraging image-wide attention, QPIC effectively captures global contextual cues essential for disambiguating complex interactions. Our method achieves a significant performance leap, including a 5.37 mAP gain on the HICO-DET benchmark and state-of-the-art results on V-COCO.

Introduction

The goal of HOI detection is to localize and recognize the relationships between humans and objects in an image. Existing methodologies generally fall into two categories: two-stage and single-stage methods. Two-stage methods first detect entities and then classify their interaction using cropped features. However, as illustrated in Figure 1, vital contextual cues often lie outside the individual bounding boxes of the human and the object. For instance, in a "washing car" interaction, the hose providing the primary visual evidence might be excluded from both the person and car crops. Single-stage methods attempt to mitigate this by using midpoint heuristics to associate humans and objects, but these often fail in crowded environments where multiple interactions overlap, leading to feature contamination.

HOI detection examples demonstrating the importance of global context. In (a), the hose providing the 'wash' cue is outside the primary bounding boxes.

QPIC addresses these limitations by utilizing a query-based approach that considers the entire image context simultaneously, treating each human-object pair as a single detection target.

Methodology

QPIC is built upon the Detection Transformer (DETR) architecture, simplifying the HOI detection pipeline by removing the need for hand-crafted components like non-maximum suppression (NMS) or anchor generation.

Feature Extraction and Global Attention

The model employs a CNN backbone (e.g., ResNet) to extract initial features, which are then processed by a Transformer Encoder. The self-attention mechanism in the encoder allows the model to aggregate information across the entire image, ensuring that distant but relevant cues are incorporated into the feature representation.

Query-Based Pairwise Detection

The core of QPIC lies in its use of $N_q$ learnable query vectors in the Transformer Decoder. Each query is designed to search the entire image for at most one human-object interaction pair. This pairwise formulation prevents the feature mixing issues common in previous single-stage methods.

The QPIC architecture: Features from a CNN backbone are processed by a Transformer encoder-decoder, with four specialized heads for pairwise detection.

Interaction Detection Heads

The output embeddings from the decoder are passed through four independent Feed-Forward Networks (FFNs) to predict the human bounding box, the object bounding box, the object class, and the action/interaction class. During training, bipartite matching via the Hungarian algorithm is used to assign predictions to ground-truth pairs, ensuring a unique and optimal mapping.

Experimental Results

QPIC was evaluated on the HICO-DET and V-COCO benchmarks, yielding substantial performance gains over previous state-of-the-art (SOTA) models.

  • HICO-DET: Using a ResNet-101 backbone, QPIC achieved 29.90 mAP, a relative improvement of over 20% (5.37 points) compared to the previous best method, DRG.
  • V-COCO: The model reached 58.8 mAP (Scenario 1), outperforming the previous SOTA by 5.6 points.

Qualitative analysis further revealed that QPIC is significantly more robust to the spatial distance between humans and objects compared to traditional CNN-based architectures, thanks to its global attention mechanism.

Conclusion

QPIC redefines HOI detection by moving from local heuristics to global, query-based set prediction. By treating human-object pairs as unified entities and leveraging the power of Transformers to capture image-wide context, it overcomes the fundamental limitations of previous localized or midpoint-based methods. Future work may extend this framework to video domains, where temporal context can be integrated similarly to spatial context.

Reference: Tamura, M., Ohashi, H., & Yoshinaga, T. (2021). QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Summary updated

BibTeX
@inproceedings{tamura2021qpic,
  author = {Masato Tamura and Hiroki Ohashi and Tomoaki Yoshinaga},
  title = {{QPIC: Query-Based Pairwise Human-Object Interaction Detection With Image-Wide Contextual Information}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01027},
}
Figure