Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restoration

Yuxiao Wang, Wenpeng Neng, Zhenao Wei, Yu Lei, Weiying Xue, Nan Zhuang, Yanwu Xu, Xinyu Jiang, Qi Liu

TL;DR

Detecting physical contact between humans and objects from a single 2D image is notoriously prone to false positives caused by accidental perspective overlap and boundary occlusion. PIHOT resolves these ambiguities by using image inpainting to reconstruct occluded object surfaces and comparing depth maps between the original and inpainted scenes. By grounding contact detection in estimated 3D spatial relationships, PIHOT sets new state-of-the-art results across standard human-object contact benchmarks.

What Problem Does This Paper Address?

Human-Object Contact (HOT) detection aims to segment the exact surface regions where a person touches an object (e.g., feet on a skateboard, hands gripping a cup). While traditional Human-Object Interaction (HOI) models classify actions high-level, HOT requires fine-grained pixel-level boundary segmentation.

Treating HOT as standard 2D semantic segmentation introduces two critical challenges:

  1. Perspective Overlap vs. True Contact: In a 2D projection, a person standing in front of a distant wall appears to overlap with it, tricking 2D models into detecting a false contact.
  2. Mutual Occlusion: When humans touch objects, their bodies frequently obscure the object’s edges and shape (e.g., a person sitting on a bench hides the bench's contour). This leaves the network with blurred, missing boundaries where the contact occurs.

Methods

PIHOT (Perspective Interaction Human-Object Contact Decector) introduces a multi-branch architecture centered around two main concepts: hallucinating hidden textures and extracting relative depth.

Input Image ───┬─► Shared Backbone ──► Contact Branch (xc) ──┐
               │                                            │
               │                                     ┌──────┴──────┐
Human Mask ──► Dilate ──► Inpainting ──► Shared Backbone ──► IPI (oa) ──► CPO ──► HOT Mask
(Expanded)                (LaMa)         (Object Branch xo) │              ▲
                                                            │              │
Depth Estimation (ZoeDepth) ────────────────────────────────┴──► SPO/IDSI ─┘
  (di: Original,  do: Repaired  ──►  ds = |di - do|)
  • Object Inpainting (OI) Module: The network dilates human body masks via 2D convolution to absorb noisy manual boundary annotations, then uses a pre-trained inpainting model (LaMa) to remove the human. This reconstructs the complete, unoccluded geometry and texture of the underlying object ($I_o$).
  • Space Perception Operation (SPO): A depth model (ZoeDepth) predicts depth maps for both the original image ($d_i$) and the inpainted object image ($d_o$). Taking the normalized absolute difference, $d_s = |d_i - d_o|$, isolates the human-object depth discrepancy—values approach zero where background elements match, and peak where the human stands in front of or touches the scene.
  • Instances Perspective Interaction (IPI): An attention mechanism where inpainted object features query original image contact features ($x_c$), focusing the network on previously occluded boundaries.
  • Instances Depth Space Interaction (IDSI) & Contact Perception Operation (CPO): Combines the depth discrepancy ($d_s$), attention-derived spatial feature ($d_a$), and contact features ($x_c, o_a$) into a unified tensor to predict the final 18-channel multi-class contact segmentation map.

Datasets

The authors evaluate PIHOT on three standard benchmarks established in recent literature:

  • HOT-Annotated: Consists of manually labeled real-world datasets spanning 15,082 images and 67,088 contact areas (combining V-COCO, HAKE, and Watch-n-Patch).
  • HOT-Generated: Synthetic/reconstructed dataset containing 20,205 images and 95,179 contact annotations derived from 3D models (PROX and SMPL-X).
  • Full Set: The combined union of both HOT-Annotated and HOT-Generated sets.

Key Insights and Ideas

  • Depth Discrepancy as a Physical Contact Gate: Subtracting the depth map of the restored background scene from the depth map of the full scene effectively creates a 3D "contact plane." If the spatial distance at the overlap region is significant, the model can decisively dismiss false-positive 2D overlaps (like walls in the background).
  • Inpainting as Structural Prior: Rather than asking the segmentation model to guess occluded boundaries directly from surrounding pixels, inpainting provides an explicit, plausible shape estimate of the hidden object before segmentation occurs.
  • Mask Dilation Matters: In practice, human segmentation ground truths have imperfect boundaries. A simple morphological dilation step ensures that slivers of the human body are not left behind as artifacts during the inpainting step.

Key Results

PIHOT consistently outperforms existing baselines, including previous state-of-the-art approaches like DHOT and generic segmentation backbones (ResNet+UperNet, ResNet+PPM):

  • On the Full Set: PIHOT reaches 77.5% Semantic Contact Accuracy (SC-Acc.) and 42.4% Contact Accuracy (C-Acc.), improving over the previous best method by 16% and 17% relative gains, alongside a 0.273 mIoU and 0.221 wIoU.
  • On HOT-Annotated: Outperforms DHOT by 11% SC-Acc. (45.3 vs. 40.7) and 14% C-Acc. (80.7 vs. 70.7).
  • On HOT-Generated: Demonstrates an even sharper boost on 3D-derived annotations, surpassing DHOT by 41% in C-Acc. (76.3 vs. 54.3) and 22% in mIoU (0.169 vs. 0.139).
  • Ablation Findings: Progressively adding OI, IPI, SPO, and IDSI yielded monotonic improvements, confirming that both object restoration and depth reasoning contribute essentially to the final performance.

Limitations and Future Work

  • Severe/Extreme Occlusion: When objects are almost entirely concealed, inpainting models may generate inaccurate geometric priors, degrading contact boundary precision.
  • Static Image Constraint: PIHOT is tailored for static 2D images and is not yet optimized for temporal consistency or real-time throughput in dynamic video streams.
  • Multi-Stage Dependencies: Because the pipeline relies on frozen, pre-trained models (LaMa for inpainting and ZoeDepth for depth estimation), future work points toward an end-to-end multi-task framework that jointly optimizes depth estimation, human parsing, and contact segmentation.

Summary updated

Abstract

Human-object contact (HOT) is designed to accurately identify the areas where humans and objects come into contact. Current methods frequently fail to account for scenarios where objects are frequently blocking the view, resulting in inaccurate identification of contact areas. To tackle this problem, we suggest using a perspective interaction HOT detector called PIHOT, which utilizes a depth map generation model to offer depth information of humans and objects related to the camera, thereby preventing false interaction detection. Furthermore, we use mask dilatation and object restoration techniques to restore the texture details in covered areas, improve the boundaries between objects, and enhance the perception of humans interacting with objects. Moreover, a spatial awareness perception is intended to concentrate on the characteristic features close to the points of contact. The experimental results show that the PIHOT algorithm achieves state-of-the-art performance on three benchmark datasets for HOT detection tasks. Compared to the most recent DHOT, our method enjoys an average improvement of 13%, 27.5%, 16%, and 18.5% on SC-Acc., C-Acc., mIoU, and wIoU metrics, respectively.
BibTeX
@inproceedings{wang2025precision,
  author = {Yuxiao Wang and Wenpeng Neng and Zhenao Wei and Yu Lei and Weiying Xue and Nan Zhuang and Yanwu Xu and Xinyu Jiang and Qi Liu},
  title = {{Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restoration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=IMEUXXbNAL},
}

Loading related content...

Figure