Abstract
Human-object contact (HOT) is designed to accurately identify the areas where humans and objects come into contact. Current methods frequently fail to account for scenarios where objects are frequently blocking the view, resulting in inaccurate identification of contact areas. To tackle this problem, we suggest using a perspective interaction HOT detector called PIHOT, which utilizes a depth map generation model to offer depth information of humans and objects related to the camera, thereby preventing false interaction detection. Furthermore, we use mask dilatation and object restoration techniques to restore the texture details in covered areas, improve the boundaries between objects, and enhance the perception of humans interacting with objects. Moreover, a spatial awareness perception is intended to concentrate on the characteristic features close to the points of contact. The experimental results show that the PIHOT algorithm achieves state-of-the-art performance on three benchmark datasets for HOT detection tasks. Compared to the most recent DHOT, our method enjoys an average improvement of 13%, 27.5%, 16%, and 18.5% on SC-Acc., C-Acc., mIoU, and wIoU metrics, respectively.
Introduction
Detecting the precise contact points between humans and objects (Human-Object Contact or HOT) is a foundational task for applications ranging from autonomous robotics to immersive virtual reality. However, current state-of-the-art methods often treat HOT detection as a 2D semantic segmentation problem. This approach suffers from two significant limitations: occlusion ambiguity, where 2D pixel overlap is mistaken for physical contact, and lost boundaries, where the human body obscures the object's edges, making precise contact definition difficult.
Methodology
The proposed PIHOT (Perspective Interaction HOT detector) framework addresses these challenges by incorporating 3D spatial awareness and texture restoration.
Object Inpainting (OI) Module
To recover obscured object geometry, PIHOT employs an Object Inpainting module. The system identifies the human subject using a dilated segmentation mask to ensure no human pixels remain, then utilizes a pre-trained LaMa inpainting model to restore the object's texture and shape. This process allows the model to "see through" the human to understand the underlying object structure.
Space Perception Operation (SPO)
The Space Perception Operation generates depth maps for both the original scene and the inpainted object-only scene. By calculating the depth disparity ($d_s = |d_i - d_o|$), the model gains a mathematical representation of the 3D distance between the human and the object, effectively disambiguating proximity from actual contact.
Interaction Mechanisms
PIHOT further enhances detection through two key interaction modules:
- Instances Perspective Interaction (IPI): A cross-attention mechanism that prioritizes features within occluded regions.
- Instances Depth Space Interaction (IDSI): This module integrates depth disparities into the feature extraction process, focusing the network on areas where spatial distance is near zero.
Experimental Results
PIHOT was evaluated on three benchmark datasets: HOT-Annotated (5,235 manually labeled images), HOT-Generated (20,205 images with 3D mesh-based labels), and a combined Full Set.
The results demonstrate that PIHOT establishes a new state-of-the-art. Compared to the previous leading model, DHOT, PIHOT achieved a 16% improvement in Semantic Contact Accuracy (SC-Acc) and a 17.5% improvement in Contact Accuracy (C-Acc) on the Full Set. Notably, on the HOT-Generated set, the model achieved a 41% improvement in C-Acc, highlighting the effectiveness of its depth-aware approach in complex 3D scenarios.
Conclusion
PIHOT significantly advances HOT detection by moving beyond 2D analysis. By leveraging depth maps to resolve spatial ambiguity and inpainting to restore obscured boundaries, the framework achieves superior precision. Future work will focus on extending these capabilities to dynamic video sequences and developing more unified end-to-end architectures for simultaneous pose and contact estimation.
Summary updated