Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restoration

Yuxiao Wang, Wenpeng Neng, Zhenao Wei, Yu Lei, Weiying Xue, Nan Zhuang, Yanwu Xu, Xinyu Jiang, Qi Liu

Abstract

Human-object contact (HOT) is designed to accurately identify the areas where humans and objects come into contact. Current methods frequently fail to account for scenarios where objects are frequently blocking the view, resulting in inaccurate identification of contact areas. To tackle this problem, we suggest using a perspective interaction HOT detector called PIHOT, which utilizes a depth map generation model to offer depth information of humans and objects related to the camera, thereby preventing false interaction detection. Furthermore, we use mask dilatation and object restoration techniques to restore the texture details in covered areas, improve the boundaries between objects, and enhance the perception of humans interacting with objects. Moreover, a spatial awareness perception is intended to concentrate on the characteristic features close to the points of contact. The experimental results show that the PIHOT algorithm achieves state-of-the-art performance on three benchmark datasets for HOT detection tasks. Compared to the most recent DHOT, our method enjoys an average improvement of 13%, 27.5%, 16%, and 18.5% on SC-Acc., C-Acc., mIoU, and wIoU metrics, respectively.

Introduction

Detecting the precise contact points between humans and objects (Human-Object Contact or HOT) is a foundational task for applications ranging from autonomous robotics to immersive virtual reality. However, current state-of-the-art methods often treat HOT detection as a 2D semantic segmentation problem. This approach suffers from two significant limitations: occlusion ambiguity, where 2D pixel overlap is mistaken for physical contact, and lost boundaries, where the human body obscures the object's edges, making precise contact definition difficult.

Methodology

The proposed PIHOT (Perspective Interaction HOT detector) framework addresses these challenges by incorporating 3D spatial awareness and texture restoration.

Object Inpainting (OI) Module

To recover obscured object geometry, PIHOT employs an Object Inpainting module. The system identifies the human subject using a dilated segmentation mask to ensure no human pixels remain, then utilizes a pre-trained LaMa inpainting model to restore the object's texture and shape. This process allows the model to "see through" the human to understand the underlying object structure.

Object restoration via inpainting: The bench is reconstructed after removing the human subjects to clarify contact boundaries.

Space Perception Operation (SPO)

The Space Perception Operation generates depth maps for both the original scene and the inpainted object-only scene. By calculating the depth disparity ($d_s = |d_i - d_o|$), the model gains a mathematical representation of the 3D distance between the human and the object, effectively disambiguating proximity from actual contact.

The PIHOT pipeline: (a) Original image with human/object detection, (b) Restored object texture, and (c) Resulting depth map used for spatial awareness.

Interaction Mechanisms

PIHOT further enhances detection through two key interaction modules:

  • Instances Perspective Interaction (IPI): A cross-attention mechanism that prioritizes features within occluded regions.
  • Instances Depth Space Interaction (IDSI): This module integrates depth disparities into the feature extraction process, focusing the network on areas where spatial distance is near zero.

Experimental Results

PIHOT was evaluated on three benchmark datasets: HOT-Annotated (5,235 manually labeled images), HOT-Generated (20,205 images with 3D mesh-based labels), and a combined Full Set.

The results demonstrate that PIHOT establishes a new state-of-the-art. Compared to the previous leading model, DHOT, PIHOT achieved a 16% improvement in Semantic Contact Accuracy (SC-Acc) and a 17.5% improvement in Contact Accuracy (C-Acc) on the Full Set. Notably, on the HOT-Generated set, the model achieved a 41% improvement in C-Acc, highlighting the effectiveness of its depth-aware approach in complex 3D scenarios.

Conclusion

PIHOT significantly advances HOT detection by moving beyond 2D analysis. By leveraging depth maps to resolve spatial ambiguity and inpainting to restore obscured boundaries, the framework achieves superior precision. Future work will focus on extending these capabilities to dynamic video sequences and developing more unified end-to-end architectures for simultaneous pose and contact estimation.

Summary updated

Abstract

Human-object contact (HOT) is designed to accurately identify the areas where humans and objects come into contact. Current methods frequently fail to account for scenarios where objects are frequently blocking the view, resulting in inaccurate identification of contact areas. To tackle this problem, we suggest using a perspective interaction HOT detector called PIHOT, which utilizes a depth map generation model to offer depth information of humans and objects related to the camera, thereby preventing false interaction detection. Furthermore, we use mask dilatation and object restoration techniques to restore the texture details in covered areas, improve the boundaries between objects, and enhance the perception of humans interacting with objects. Moreover, a spatial awareness perception is intended to concentrate on the characteristic features close to the points of contact. The experimental results show that the PIHOT algorithm achieves state-of-the-art performance on three benchmark datasets for HOT detection tasks. Compared to the most recent DHOT, our method enjoys an average improvement of 13%, 27.5%, 16%, and 18.5% on SC-Acc., C-Acc., mIoU, and wIoU metrics, respectively.
BibTeX
@inproceedings{wang2025precision,
  author = {Yuxiao Wang and Wenpeng Neng and Zhenao Wei and Yu Lei and Weiying Xue and Nan Zhuang and Yanwu Xu and Xinyu Jiang and Qi Liu},
  title = {{Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restoration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=IMEUXXbNAL},
}
Figure