Geometric Features Enhanced Human-Object Interaction Detection

Manli Zhu, Edmond S. L. Ho, Shuang Chen, Longzhi Yang, Hubert P. H. Shum

Abstract

Human-Object Interaction (HOI) detection is a critical task in computer vision that involves identifying triplets of <human, interaction, object>. Traditional methods often struggle with occlusion and the coarse nature of bounding box representations. We present GeoHOI, a novel Transformer-based framework that enhances HOI detection by integrating fine-grained geometric keypoints. By employing a self-supervised keypoint learner called UniPointNet, GeoHOI captures structural relationships that standard models miss. Our approach demonstrates significant performance improvements on standard benchmarks, particularly in complex, real-world scenarios.

Introduction: The Challenge of Occlusion and Coarse Data

Detecting interactions like <human, riding, bicycle> requires more than just identifying objects; it requires understanding the spatial and structural nuances of their relationship. Modern Transformer-based models typically rely on visual features from RGB images and simple bounding boxes. However, this "coarse" data approach fails when objects are partially occluded—for instance, a person sitting behind a desk. Bounding boxes are often too blunt to capture the precise points of contact, such as a hand on a laptop or a foot on a shovel, which are vital for accurate interaction classification. GeoHOI addresses this by introducing fine-grained geometric priors (keypoints) into the end-to-end Transformer pipeline.

Methodology

The GeoHOI framework enhances the Structure-aware Transformer over Interaction Proposals (STIP) architecture through three primary components:

  1. UniPointNet: This self-supervised module detects keypoints across any object category. To ensure robustness against varying appearances, it operates on binary segmentation masks rather than RGB images. This design choice forces the network to learn the underlying shape and structural landmarks (like corners or edges) regardless of the object's color or texture.
  2. Keypoint-aware Interactiveness Prediction (KIP): Utilizing a Graph Convolutional Network (GCN), KIP models the holistic relationships between human and object keypoints. It filters out non-interactive pairs by calculating the likelihood of interaction based on geometric proximity and structural alignment.
  3. Part Attention Module (PAM): For identified interactive pairs, PAM applies self-attention to local image patches centered around the detected keypoints. This allows the model to focus on the most informative "parts" of the interaction, such as the specific point where a hand grips a handle, to refine the final classification.
Overview of the GeoHOI framework, illustrating the integration of keypoint detection, interactiveness prediction, and part-level attention.

Key Results and Evaluation

GeoHOI was evaluated on the V-COCO and HICO-DET benchmarks, as well as a specialized Post-Disaster Dataset (PDD) for real-world rescue applications.

  • State-of-the-Art Performance: On V-COCO, GeoHOI achieved a mean Average Precision (mAP) of 69.4, outperforming its backbone STIP model by 3.4 points and surpassing other recent methods like MUREN and ViPLO.
  • HICO-DET Success: The model achieved a competitive 37.12 mAP in the "Known Object" setting. Notably, it showed significant gains in "Rare" interaction categories, where geometric patterns remain consistent even when visual data is scarce.
  • Real-World Utility: In post-disaster scenarios, GeoHOI outperformed standard detection models in identifying humans amidst rubble by leveraging interaction cues (e.g., a person "lying on" the ground).
Performance comparison of GeoHOI against state-of-the-art methods on V-COCO and HICO-DET benchmarks.

Discussion and Future Directions

Ablation studies revealed that 32 keypoints per instance provide the optimal balance between capturing shape detail and avoiding overfitting. While geometric features act as a stabilizer under heavy occlusion, the model does face limitations. The UniPointNet module can struggle with very thin objects (e.g., baseball bats) due to the low pixel count in their masks. Additionally, the added complexity increases inference time to approximately 588ms per image, making it less suitable for high-speed real-time applications without further optimization.

Future work may involve integrating Large Language Models (LLMs) or vision-language models like CLIP to enhance the recognition of "zero-shot" or extremely rare interactions by leveraging broader semantic knowledge.

Summary updated

BibTeX
@article{zhu2024geometric,
  author = {Manli Zhu and Edmond S. L. Ho and Shuang Chen and Longzhi Yang and Hubert P. H. Shum},
  title = {{Geometric Features Enhanced Human-Object Interaction Detection}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2024},
  doi = {10.1109/tim.2024.3427800},
}
Figure