Geometric Features Enhanced Human-Object Interaction Detection

Manli Zhu, Edmond S. L. Ho, Shuang Chen, Longzhi Yang, Hubert P. H. Shum

TL;DR

GeoHOI is a novel Transformer-based framework that boosts Human-Object Interaction (HOI) detection by integrating fine-grained geometric keypoints. By utilizing a new self-supervised keypoint learner called UniPointNet, the model captures structural relationships that standard bounding boxes miss, significantly improving performance in complex, occluded real-world scenes.

What Problem Does This Paper Address?

Detecting Human-Object Interactions (HOI) involves identifying triplets of <human, interaction, object> within an image (e.g., <human, riding, bicycle>). While modern Transformer-based models have made great strides, they often rely on "coarse" data: visual features from RGB images and simple bounding boxes.

This approach hits a wall when occlusion occurs. If a person is partially hidden behind a desk, visual cues become unreliable. Furthermore, bounding boxes are too blunt to capture the nuances of an interaction—knowing exactly where a hand touches a laptop or how a foot rests on a shovel is critical for accurate classification. The authors aim to bridge this gap by introducing fine-grained geometric priors (keypoints) into the end-to-end Transformer pipeline.

Methods

The authors propose GeoHOI, which enhances the existing STIP (Structure-aware Transformer over Interaction Proposals) architecture with three key components:

  1. UniPointNet: A novel self-supervised module designed to detect keypoints across any object category, not just humans. To make it robust against varying appearances, it operates on binary segmentation masks rather than RGB images. This forces the network to learn the underlying shape and structure of objects.
  2. Keypoint-aware Interactiveness Prediction (KIP): This module uses a Graph Convolutional Network (GCN) to model the holistic relationships between human and object keypoints. It calculates the likelihood of an interaction based on geometric proximity and structural alignment, effectively filtering out non-interactive pairs.
  3. Part Attention Module (PAM): Once an interactive pair is identified, PAM uses self-attention on local image patches centered around the detected keypoints. This allows the model to focus on the most informative "parts" of the interaction (like a hand on a handle) to refine the final classification.

Datasets

The model was evaluated on two standard HOI benchmarks and one specialized real-world dataset:

  • V-COCO: A subset of MS-COCO with 29 action classes.
  • HICO-DET: A large-scale dataset featuring 600 interaction types across 80 object categories.
  • PDD (Post-Disaster Dataset): A dataset of images from real-world ruin scenes captured by UAVs, used for a case study on rescue applications.

Key Insights and Ideas

  • Masks Over Pixels: One of the most clever moves is training UniPointNet on binary masks. By stripping away color and texture, the model becomes category-agnostic. It learns that a "keypoint" is a structural landmark (like a corner or an edge) regardless of whether it's on a chair or a dog.
  • Geometry as a Stabilizer: Geometric features are inherently more stable than visual ones under heavy occlusion. Even if a person's torso is hidden, the geometric relationship between their visible hand and a cup provides a strong structural prior for "drinking."
  • The "Sweet Spot" for Detail: Through ablation studies, the authors found that 32 keypoints per instance provided the best balance. Fewer keypoints failed to capture enough shape detail, while more led to overfitting and noise.

Key Results

  • State-of-the-Art Performance: On the V-COCO dataset, GeoHOI achieved 69.4 mAP, outperforming the backbone STIP model by 3.4 points and surpassing other recent methods like MUREN and ViPLO.
  • HICO-DET Success: It achieved a competitive 37.12 mAP in the "Known Object" setting, showing significant gains in detecting "Rare" interaction categories where visual data is scarce but geometric patterns are consistent.
  • Real-World Utility: In the post-disaster rescue case study, GeoHOI outperformed standard detection-based models in identifying humans in complex rubble, proving that understanding interactions (like a person "holding" a pet or "lying on" ground) helps refine human localization.

Limitations and Future Work

  • Computational Complexity: The addition of keypoint detection and GCNs increases the computational load. GeoHOI's inference time is roughly 588ms per image, compared to STIP's 117ms. While still functional, it is less suitable for high-speed real-time applications without further optimization.
  • Small/Slender Objects: The UniPointNet module struggles with very thin objects (e.g., baseball bats, tennis rackets) because their masks have very few pixels, leading to noisy keypoint detection.
  • Future Directions: The authors suggest exploring Large Language Models (LLMs) or vision-language models like CLIP to help the system recognize "zero-shot" or extremely rare interactions by leveraging semantic knowledge.

Summary updated

Abstract

Cameras are essential vision instruments to capture images for pattern detection and measurement. Human–object interaction (HOI) detection is one of the most popular pattern detection approaches for captured human-centric visual scenes. Recently, Transformer-based models have become the dominant approach for HOI detection due to their advanced network architectures and, thus, promising results. However, most of them follow the one-stage design of vanilla Transformer, leaving rich geometric priors underexploited and leading to compromised performance, especially when occlusion occurs. Given that geometric features tend to outperform visual ones in occluded scenarios and offer information that complements visual cues, we propose a novel end-to-end Transformer-style HOI detection model, i.e., geometric features enhanced HOI detector (GeoHOI). One key part of the model is a new unified self-supervised keypoint learning method named UniPointNet that bridges the gap of consistent keypoint representation across diverse object categories, including humans. GeoHOI effectively upgrades a Transformer-based HOI detector benefiting from the keypoints similarities measuring the likelihood of HOIs and local keypoint patches to enhance interaction query representation, so as to boost HOI predictions. Extensive experiments show that the proposed method outperforms the state-of-the-art models on V-COCO and achieves competitive performance on HICO-DET. Case study results on the postdisaster rescue with vision-based instruments showcase the applicability of the proposed GeoHOI in real-world applications.
BibTeX
@article{zhu2024geometric,
  author = {Manli Zhu and Edmond S. L. Ho and Shuang Chen and Longzhi Yang and Hubert P. H. Shum},
  title = {{Geometric Features Enhanced Human-Object Interaction Detection}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2024},
  doi = {10.1109/tim.2024.3427800},
}
Figure