TL;DR
Reconstructing 3D scenes from a single 2D image typically leads to mesh collisions, depth-ordering errors, and poor object pose estimates when humans heavily occlude objects. This paper introduces an optimization-based framework without 3D supervision that models both human–human and human–object interactions using a novel collision loss and leverages generative inpainting to reveal occluded objects for robust 6-DoF pose estimation.
What Problem Does This Paper Address?
Estimating the 3D poses and shapes of people and objects from a single monocular image is inherently ill-posed. Existing methods often estimate people and objects in isolation or consider only human–object interactions (like PHOSA) while ignoring interactions between multiple people. Consequently, independent reconstructions place meshes directly inside each other (interpenetration/collision) or swap foreground and background depth orders.
Furthermore, when people occlude objects (e.g., sitting on a bench or riding a bike), 2D object segmentations are heavily truncated, causing standard silhouette-fitting methods to predict incorrect object sizes and 6-DoF poses. The authors set out to produce a globally coherent 3D scene from an unconstrained in-the-wild image without requiring scene-level or object-level 3D supervision.
Methods
The pipeline operates in three main stages:
- Independent Human & Object Initialization:
- Humans: Extracted using Mask R-CNN and fitted with the parametric SMPL model (via EFT/Joo et al.) to predict pose $\theta$, shape $\beta$, and initial camera translation.
- Objects: Uses pre-selected CAD exemplar meshes (from Free3D / 3D-RCNN).
- Inpainting for Occlusion-Aware 6-DoF Object Fitting:
- To prevent occluded object silhouettes from corrupting 3D object fitting, the method checks pairwise 2D bounding box Intersection over Union (IoU).
- If IoU $> 0.3$, the occluding person/object is masked out and removed using a deep inpainting model (EdgeConnect).
- A segmentation model (PointRend) runs on the inpainted image to recover the complete object mask. The 6-DoF object pose is then solved via differentiable rendering using an occlusion-aware silhouette loss augmented with a chamfer distance loss.
- Multi-Stage Optimization with Interaction Constraints:
- Human Pose Optimization: Interacting humans (identified via 3D bounding box proximity) are refined using a novel Human–Human Collision Loss ($L_{H-collision}$, penalizing interpenetration using truncated voxel Signed Distance Fields [SDFs]), a depth-ordering loss, and a proximity interaction loss.
- Joint Human–Object Optimization: Jointly optimizes human translation/scale and object translation/rotation/scale ($L_{\text{joint-loss}}$) combining human–human losses, human–object collision losses, coarse/fine contact interaction priors, and silhouette alignment.
Datasets
- COCO-2017: Used for evaluating complex multi-person and human–object interactions in in-the-wild images.
- In-the-wild Internet Images: Additional qualitative testing on images sourced from YouTube and Google.
- Note: No newly created benchmark dataset was introduced. Evaluation relies on test images from COCO-2017.
Key Insights and Ideas
- Inpainting "unhides" objects for differentiable rendering: Fitting 3D models to truncated 2D masks often forces CAD models to collapse or rotate unnaturally. Generative inpainting hallucinating the missing object parts yields substantially cleaner silhouette supervision.
- Voxel SDFs per instance avoid scene explosion: Rather than computing a massive, memory-heavy voxel grid for the entire 3D scene, the collision loss computes local, tightly bounded signed distance grids for each person, making pairwise mesh interpenetration penalties differentiable and computationally feasible.
- Human–human interactions constrain scene depth: Modeling human–object interactions alone often distorts the relative spatial relations between people. Co-optimizing human–human and human–object constraints ensures globally consistent scenes.
Key Results
The authors evaluated against state-of-the-art baselines: PHOSA (human–object reconstruction) as well as ROMP and BEV (multi-person human reconstruction).
- Mesh Collision Metrics:
- Human–human collision error ($E_{H-col}$): Dropped from 196.42 (PHOSA) and 106.25 (BEV) down to 16.46 (Ours) — an approximate 80% reduction.
- Human–object collision error ($E_{HO-col}$): Reduced from 257.21 (PHOSA) to 26.65 (Ours).
- Depth Ordering Disparity:
- Human–human depth error ($E_{H-depth}$) decreased from 241.68 (PHOSA) to 54.37 (Ours).
- Human–object depth error ($E_{HO-depth}$) dropped from 140.84 (PHOSA) to 73.77 (Ours).
- User Study: In forced-choice perceptual evaluations, human judges preferred the proposed reconstructions over PHOSA (88% of the time), ROMP (80%), and BEV (74%).
- Ablation Findings: Removing collision loss ($L_{\text{collision}}$) or occlusion-aware silhouette loss ($L_{occ-sil}$) caused the sharpest drops in visual quality (preferred over ablations 83% and 77% of the time, respectively).
Limitations and Future Work
- Reliance on Fixed Exemplar Meshes: Object reconstruction relies on selecting pre-existing rigid CAD models rather than reconstructing arbitrary or deformable 3D object shapes.
- Multi-Stage Optimization Latency: The framework relies on iterative gradient-based optimization loops (e.g., up to 500 iterations for joint fitting) along with multiple neural submodules (Mask R-CNN, EdgeConnect, PointRend, differentiable rendering), which makes real-time inference infeasible.
- Heuristic Interaction Detection: Interaction pairs are determined using axis-aligned 3D bounding box overlaps and IoU thresholds ($>0.3$), which may miss subtle or non-overlapping physical interactions.
Summary updated