SINGLE-IMAGE COHERENT RECONSTRUCTION OF OBJECTS AND HUMANS

sarthak batra, Partha Pratim Chakrabarti, Simon Hadfield, Armin MustafaSubmission to ICLR 2024 · Rejected

TL;DR

Reconstructing 3D scenes from a single 2D image typically leads to mesh collisions, depth-ordering errors, and poor object pose estimates when humans heavily occlude objects. This paper introduces an optimization-based framework without 3D supervision that models both human–human and human–object interactions using a novel collision loss and leverages generative inpainting to reveal occluded objects for robust 6-DoF pose estimation.

What Problem Does This Paper Address?

Estimating the 3D poses and shapes of people and objects from a single monocular image is inherently ill-posed. Existing methods often estimate people and objects in isolation or consider only human–object interactions (like PHOSA) while ignoring interactions between multiple people. Consequently, independent reconstructions place meshes directly inside each other (interpenetration/collision) or swap foreground and background depth orders.

Furthermore, when people occlude objects (e.g., sitting on a bench or riding a bike), 2D object segmentations are heavily truncated, causing standard silhouette-fitting methods to predict incorrect object sizes and 6-DoF poses. The authors set out to produce a globally coherent 3D scene from an unconstrained in-the-wild image without requiring scene-level or object-level 3D supervision.

Methods

The pipeline operates in three main stages:

  1. Independent Human & Object Initialization:
  2. Humans: Extracted using Mask R-CNN and fitted with the parametric SMPL model (via EFT/Joo et al.) to predict pose $\theta$, shape $\beta$, and initial camera translation.
  3. Objects: Uses pre-selected CAD exemplar meshes (from Free3D / 3D-RCNN).
  4. Inpainting for Occlusion-Aware 6-DoF Object Fitting:
  5. To prevent occluded object silhouettes from corrupting 3D object fitting, the method checks pairwise 2D bounding box Intersection over Union (IoU).
  6. If IoU $> 0.3$, the occluding person/object is masked out and removed using a deep inpainting model (EdgeConnect).
  7. A segmentation model (PointRend) runs on the inpainted image to recover the complete object mask. The 6-DoF object pose is then solved via differentiable rendering using an occlusion-aware silhouette loss augmented with a chamfer distance loss.
  8. Multi-Stage Optimization with Interaction Constraints:
  9. Human Pose Optimization: Interacting humans (identified via 3D bounding box proximity) are refined using a novel Human–Human Collision Loss ($L_{H-collision}$, penalizing interpenetration using truncated voxel Signed Distance Fields [SDFs]), a depth-ordering loss, and a proximity interaction loss.
  10. Joint Human–Object Optimization: Jointly optimizes human translation/scale and object translation/rotation/scale ($L_{\text{joint-loss}}$) combining human–human losses, human–object collision losses, coarse/fine contact interaction priors, and silhouette alignment.

Datasets

  • COCO-2017: Used for evaluating complex multi-person and human–object interactions in in-the-wild images.
  • In-the-wild Internet Images: Additional qualitative testing on images sourced from YouTube and Google.
  • Note: No newly created benchmark dataset was introduced. Evaluation relies on test images from COCO-2017.

Key Insights and Ideas

  • Inpainting "unhides" objects for differentiable rendering: Fitting 3D models to truncated 2D masks often forces CAD models to collapse or rotate unnaturally. Generative inpainting hallucinating the missing object parts yields substantially cleaner silhouette supervision.
  • Voxel SDFs per instance avoid scene explosion: Rather than computing a massive, memory-heavy voxel grid for the entire 3D scene, the collision loss computes local, tightly bounded signed distance grids for each person, making pairwise mesh interpenetration penalties differentiable and computationally feasible.
  • Human–human interactions constrain scene depth: Modeling human–object interactions alone often distorts the relative spatial relations between people. Co-optimizing human–human and human–object constraints ensures globally consistent scenes.

Key Results

The authors evaluated against state-of-the-art baselines: PHOSA (human–object reconstruction) as well as ROMP and BEV (multi-person human reconstruction).

  • Mesh Collision Metrics:
  • Human–human collision error ($E_{H-col}$): Dropped from 196.42 (PHOSA) and 106.25 (BEV) down to 16.46 (Ours) — an approximate 80% reduction.
  • Human–object collision error ($E_{HO-col}$): Reduced from 257.21 (PHOSA) to 26.65 (Ours).
  • Depth Ordering Disparity:
  • Human–human depth error ($E_{H-depth}$) decreased from 241.68 (PHOSA) to 54.37 (Ours).
  • Human–object depth error ($E_{HO-depth}$) dropped from 140.84 (PHOSA) to 73.77 (Ours).
  • User Study: In forced-choice perceptual evaluations, human judges preferred the proposed reconstructions over PHOSA (88% of the time), ROMP (80%), and BEV (74%).
  • Ablation Findings: Removing collision loss ($L_{\text{collision}}$) or occlusion-aware silhouette loss ($L_{occ-sil}$) caused the sharpest drops in visual quality (preferred over ablations 83% and 77% of the time, respectively).

Limitations and Future Work

  • Reliance on Fixed Exemplar Meshes: Object reconstruction relies on selecting pre-existing rigid CAD models rather than reconstructing arbitrary or deformable 3D object shapes.
  • Multi-Stage Optimization Latency: The framework relies on iterative gradient-based optimization loops (e.g., up to 500 iterations for joint fitting) along with multiple neural submodules (Mask R-CNN, EdgeConnect, PointRend, differentiable rendering), which makes real-time inference infeasible.
  • Heuristic Interaction Detection: Interaction pairs are determined using axis-aligned 3D bounding box overlaps and IoU thresholds ($>0.3$), which may miss subtle or non-overlapping physical interactions.

Summary updated

Abstract

Existing methods for reconstruction of objects and humans from a monocular image suffer from severe mesh collisions and performance limitations for interacting occluding objects. In this paper, we introduce a method that deduces spatial configurations and achieves globally consistent 3D reconstruction for interacting objects and people captured within a single image. Our contributions encompass: 1) an optimization framework, featuring a novel collision loss, tailored to handle complex human-object and human-human interactions, ensuring spatially coherent scene reconstruction; and 2) a novel technique for robustly estimating 6 degrees of freedom (DOF) poses, particularly for heavily occluded objects, exploiting image inpainting. Notably, our proposed method operates effectively on images from real-world scenarios, without necessitating scene or object-level 3D supervision. Through both qualitative and quantitative assessments, we demonstrate the superior quality of our reconstructions, showcasing a significant reduction in collisions in scenes with multiple interacting humans and objects.
BibTeX
@misc{batra2024single,
  author = {sarthak batra and Partha Pratim Chakrabarti and Simon Hadfield and Armin Mustafa},
  title = {{SINGLE-IMAGE COHERENT RECONSTRUCTION OF OBJECTS AND HUMANS}},
  year = {2024},
  eprint = {2408.08086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.08086},
}
Figure