TL;DR
GeoHOI is a novel Transformer-based framework that boosts Human-Object Interaction (HOI) detection by integrating fine-grained geometric keypoints. By utilizing a new self-supervised keypoint learner called UniPointNet, the model captures structural relationships that standard bounding boxes miss, significantly improving performance in complex, occluded real-world scenes.
What Problem Does This Paper Address?
Detecting Human-Object Interactions (HOI) involves identifying triplets of <human, interaction, object> within an image (e.g., <human, riding, bicycle>). While modern Transformer-based models have made great strides, they often rely on "coarse" data: visual features from RGB images and simple bounding boxes.
This approach hits a wall when occlusion occurs. If a person is partially hidden behind a desk, visual cues become unreliable. Furthermore, bounding boxes are too blunt to capture the nuances of an interaction—knowing exactly where a hand touches a laptop or how a foot rests on a shovel is critical for accurate classification. The authors aim to bridge this gap by introducing fine-grained geometric priors (keypoints) into the end-to-end Transformer pipeline.
Methods
The authors propose GeoHOI, which enhances the existing STIP (Structure-aware Transformer over Interaction Proposals) architecture with three key components:
- UniPointNet: A novel self-supervised module designed to detect keypoints across any object category, not just humans. To make it robust against varying appearances, it operates on binary segmentation masks rather than RGB images. This forces the network to learn the underlying shape and structure of objects.
- Keypoint-aware Interactiveness Prediction (KIP): This module uses a Graph Convolutional Network (GCN) to model the holistic relationships between human and object keypoints. It calculates the likelihood of an interaction based on geometric proximity and structural alignment, effectively filtering out non-interactive pairs.
- Part Attention Module (PAM): Once an interactive pair is identified, PAM uses self-attention on local image patches centered around the detected keypoints. This allows the model to focus on the most informative "parts" of the interaction (like a hand on a handle) to refine the final classification.
Datasets
The model was evaluated on two standard HOI benchmarks and one specialized real-world dataset:
- V-COCO: A subset of MS-COCO with 29 action classes.
- HICO-DET: A large-scale dataset featuring 600 interaction types across 80 object categories.
- PDD (Post-Disaster Dataset): A dataset of images from real-world ruin scenes captured by UAVs, used for a case study on rescue applications.
Key Insights and Ideas
- Masks Over Pixels: One of the most clever moves is training UniPointNet on binary masks. By stripping away color and texture, the model becomes category-agnostic. It learns that a "keypoint" is a structural landmark (like a corner or an edge) regardless of whether it's on a chair or a dog.
- Geometry as a Stabilizer: Geometric features are inherently more stable than visual ones under heavy occlusion. Even if a person's torso is hidden, the geometric relationship between their visible hand and a cup provides a strong structural prior for "drinking."
- The "Sweet Spot" for Detail: Through ablation studies, the authors found that 32 keypoints per instance provided the best balance. Fewer keypoints failed to capture enough shape detail, while more led to overfitting and noise.
Key Results
- State-of-the-Art Performance: On the V-COCO dataset, GeoHOI achieved 69.4 mAP, outperforming the backbone STIP model by 3.4 points and surpassing other recent methods like MUREN and ViPLO.
- HICO-DET Success: It achieved a competitive 37.12 mAP in the "Known Object" setting, showing significant gains in detecting "Rare" interaction categories where visual data is scarce but geometric patterns are consistent.
- Real-World Utility: In the post-disaster rescue case study, GeoHOI outperformed standard detection-based models in identifying humans in complex rubble, proving that understanding interactions (like a person "holding" a pet or "lying on" ground) helps refine human localization.
Limitations and Future Work
- Computational Complexity: The addition of keypoint detection and GCNs increases the computational load. GeoHOI's inference time is roughly 588ms per image, compared to STIP's 117ms. While still functional, it is less suitable for high-speed real-time applications without further optimization.
- Small/Slender Objects: The UniPointNet module struggles with very thin objects (e.g., baseball bats, tennis rackets) because their masks have very few pixels, leading to noisy keypoint detection.
- Future Directions: The authors suggest exploring Large Language Models (LLMs) or vision-language models like CLIP to help the system recognize "zero-shot" or extremely rare interactions by leveraging semantic knowledge.
Summary updated