TL;DR
Detecting physical contact between humans and objects from a single 2D image is notoriously prone to false positives caused by accidental perspective overlap and boundary occlusion. PIHOT resolves these ambiguities by using image inpainting to reconstruct occluded object surfaces and comparing depth maps between the original and inpainted scenes. By grounding contact detection in estimated 3D spatial relationships, PIHOT sets new state-of-the-art results across standard human-object contact benchmarks.
What Problem Does This Paper Address?
Human-Object Contact (HOT) detection aims to segment the exact surface regions where a person touches an object (e.g., feet on a skateboard, hands gripping a cup). While traditional Human-Object Interaction (HOI) models classify actions high-level, HOT requires fine-grained pixel-level boundary segmentation.
Treating HOT as standard 2D semantic segmentation introduces two critical challenges:
- Perspective Overlap vs. True Contact: In a 2D projection, a person standing in front of a distant wall appears to overlap with it, tricking 2D models into detecting a false contact.
- Mutual Occlusion: When humans touch objects, their bodies frequently obscure the object’s edges and shape (e.g., a person sitting on a bench hides the bench's contour). This leaves the network with blurred, missing boundaries where the contact occurs.
Methods
PIHOT (Perspective Interaction Human-Object Contact Decector) introduces a multi-branch architecture centered around two main concepts: hallucinating hidden textures and extracting relative depth.
Input Image ───┬─► Shared Backbone ──► Contact Branch (xc) ──┐
│ │
│ ┌──────┴──────┐
Human Mask ──► Dilate ──► Inpainting ──► Shared Backbone ──► IPI (oa) ──► CPO ──► HOT Mask
(Expanded) (LaMa) (Object Branch xo) │ ▲
│ │
Depth Estimation (ZoeDepth) ────────────────────────────────┴──► SPO/IDSI ─┘
(di: Original, do: Repaired ──► ds = |di - do|)
- Object Inpainting (OI) Module: The network dilates human body masks via 2D convolution to absorb noisy manual boundary annotations, then uses a pre-trained inpainting model (LaMa) to remove the human. This reconstructs the complete, unoccluded geometry and texture of the underlying object ($I_o$).
- Space Perception Operation (SPO): A depth model (ZoeDepth) predicts depth maps for both the original image ($d_i$) and the inpainted object image ($d_o$). Taking the normalized absolute difference, $d_s = |d_i - d_o|$, isolates the human-object depth discrepancy—values approach zero where background elements match, and peak where the human stands in front of or touches the scene.
- Instances Perspective Interaction (IPI): An attention mechanism where inpainted object features query original image contact features ($x_c$), focusing the network on previously occluded boundaries.
- Instances Depth Space Interaction (IDSI) & Contact Perception Operation (CPO): Combines the depth discrepancy ($d_s$), attention-derived spatial feature ($d_a$), and contact features ($x_c, o_a$) into a unified tensor to predict the final 18-channel multi-class contact segmentation map.
Datasets
The authors evaluate PIHOT on three standard benchmarks established in recent literature:
- HOT-Annotated: Consists of manually labeled real-world datasets spanning 15,082 images and 67,088 contact areas (combining V-COCO, HAKE, and Watch-n-Patch).
- HOT-Generated: Synthetic/reconstructed dataset containing 20,205 images and 95,179 contact annotations derived from 3D models (PROX and SMPL-X).
- Full Set: The combined union of both HOT-Annotated and HOT-Generated sets.
Key Insights and Ideas
- Depth Discrepancy as a Physical Contact Gate: Subtracting the depth map of the restored background scene from the depth map of the full scene effectively creates a 3D "contact plane." If the spatial distance at the overlap region is significant, the model can decisively dismiss false-positive 2D overlaps (like walls in the background).
- Inpainting as Structural Prior: Rather than asking the segmentation model to guess occluded boundaries directly from surrounding pixels, inpainting provides an explicit, plausible shape estimate of the hidden object before segmentation occurs.
- Mask Dilation Matters: In practice, human segmentation ground truths have imperfect boundaries. A simple morphological dilation step ensures that slivers of the human body are not left behind as artifacts during the inpainting step.
Key Results
PIHOT consistently outperforms existing baselines, including previous state-of-the-art approaches like DHOT and generic segmentation backbones (ResNet+UperNet, ResNet+PPM):
- On the Full Set: PIHOT reaches 77.5% Semantic Contact Accuracy (SC-Acc.) and 42.4% Contact Accuracy (C-Acc.), improving over the previous best method by 16% and 17% relative gains, alongside a 0.273 mIoU and 0.221 wIoU.
- On HOT-Annotated: Outperforms DHOT by 11% SC-Acc. (45.3 vs. 40.7) and 14% C-Acc. (80.7 vs. 70.7).
- On HOT-Generated: Demonstrates an even sharper boost on 3D-derived annotations, surpassing DHOT by 41% in C-Acc. (76.3 vs. 54.3) and 22% in mIoU (0.169 vs. 0.139).
- Ablation Findings: Progressively adding OI, IPI, SPO, and IDSI yielded monotonic improvements, confirming that both object restoration and depth reasoning contribute essentially to the final performance.
Limitations and Future Work
- Severe/Extreme Occlusion: When objects are almost entirely concealed, inpainting models may generate inaccurate geometric priors, degrading contact boundary precision.
- Static Image Constraint: PIHOT is tailored for static 2D images and is not yet optimized for temporal consistency or real-time throughput in dynamic video streams.
- Multi-Stage Dependencies: Because the pipeline relies on frozen, pre-trained models (LaMa for inpainting and ZoeDepth for depth estimation), future work points toward an end-to-end multi-task framework that jointly optimizes depth estimation, human parsing, and contact segmentation.
Summary updated