Abstract
Existing methods for reconstructing objects and humans from a monocular image suffer from severe mesh collisions and performance limitations for interacting occluding objects. This paper introduces a method to obtain a globally consistent 3D reconstruction of interacting objects and people from a single image. Our contributions include: 1) an optimization framework, featuring a collision loss, tailored to handle human-object and human-human interactions, ensuring spatially coherent scene reconstruction; and 2) a novel technique to robustly estimate 6 degrees of freedom (DOF) poses, specifically for heavily occluded objects, exploiting image inpainting. Notably, our proposed method operates effectively on images from real-world scenarios, without necessitating scene or object-level 3D supervision. Extensive qualitative and quantitative evaluation against existing methods demonstrates a significant reduction in collisions in the final reconstructions of scenes with multiple interacting humans and objects and a more coherent scene reconstruction.
Introduction
Reconstructing a 3D scene from a single 2D image is a fundamentally ill-posed problem, as infinite 3D configurations can produce the same 2D projection. Current state-of-the-art methods typically reconstruct humans and objects independently, which often results in "ghosting" and physical impossibilities. When these independent models are placed in a shared 3D scene, they frequently occupy the same space or appear in incorrect depth orders. While previous frameworks like PHOSA attempted to address human-object pairs, they struggled with multiple interacting humans and failed when objects were heavily occluded. This research aims to provide "spatial common sense" to 3D reconstruction by ensuring global consistency.
Methodology
The proposed method utilizes a multi-stage optimization pipeline that operates without explicit 3D supervision, learning instead from 2D images.
Optimization Framework
The process begins with independent initial estimations for humans and objects using established models. To resolve human-human interactions, the framework introduces a Collision Loss based on a modified Signed Distance Field (SDF). This loss function penalizes overlapping human meshes, pushing them apart during optimization. Additionally, a Depth-Ordering Loss is employed to maintain the relative depth positions observed in the 2D image.
Occlusion-Aware Object Pose
A key innovation is the use of image inpainting to handle heavy occlusions. When a person blocks an object (e.g., a bicycle), the resulting incomplete segmentation mask leads to poor 3D fitting. By using an image inpainting network to "remove" the occluding person and hallucinate the missing parts of the object, the system can perform 3D object fitting against a refined, complete mask. Finally, a joint optimization step ties all entities together, scoring the layout based on orientations, collisions, and shape exemplars.
Key Insights
The framework treats inpainting as a geometric tool rather than just a photo editing technique. By filling in missing parts of occluded objects, 6-DOF pose estimation becomes significantly more accurate. Furthermore, in the absence of 3D labels, physical constraints—such as the principle that two objects cannot occupy the same space—act as a powerful supervisory signal. This holistic approach recognizes that depth ambiguities are best resolved by considering the interactions between all entities in a scene.
Results and Discussion
Evaluation was primarily conducted on the COCO-2017 dataset. Since COCO lacks 3D ground truth for "in-the-wild" scenes, the authors relied on qualitative comparisons and user studies. Robustness was further tested on complex, crowded scenes sourced from YouTube and Google Images.
Quantitative analysis shows a dramatic reduction in human-human collisions ($E_{H-col}$), dropping from 79.42 (PHOSA) to 16.46.
In a forced-choice user study, participants preferred the proposed reconstructions over PHOSA 88% of the time. Qualitatively, the results show marked improvements in spatial arrangement; objects that previously appeared to float or merge into people are correctly placed on the ground or held in hands.
Conclusion
While the optimization-based approach is computationally slower than feed-forward networks, it provides a significant leap in reconstruction coherence. Current limitations include a focus on coarse interactions rather than fine-grained details like finger placement. Future work will explore incorporating fine-grained interaction priors to further refine human-object poses.
Summary updated