Abstract
Generating 3D scenes from a single RGB image remains a significant challenge due to object occlusion and depth ambiguity. We propose a novel three-stage framework that ensures both high-quality object generation and scene coherence. Our method begins with instance segmentation and inpainting to recover occluded details, followed by 3D asset generation. We then estimate scene geometry via pseudo-stereo viewpoints to select the best-matching models. Finally, we optimize the spatial layout by minimizing Chamfer distance in both 3D and 2D projection spaces. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in geometric accuracy, texture fidelity, and layout synthesis.
Introduction
While AI has made significant progress in generating single 3D objects, creating entire multi-object scenes from a single viewpoint remains difficult. Current methods often produce "melted" geometry or fail to place objects correctly in 3D space due to flawed monocular depth estimates. This paper introduces a "divide-and-conquer" framework that decouples object generation from layout optimization to maintain high-fidelity details while ensuring spatial coherence.
Methodology
Our framework follows a three-stage "decomposition-recomposition" pipeline designed to handle the complexities of multi-object scenes.
1. Instance Segmentation and Generation
The system first detects and segments individual objects using Grounding DINO and SAM. To handle occlusions, a Vision-Language Model (GPT-4o) inpaints missing parts of the 2D segments. These complete images are then used by a 3D generator (Trellis) to create candidate meshes for each object.
2. Point Cloud Extraction & Model Selection
Using DUSt3R, the framework creates a pseudo-stereo pair from the input image to estimate camera parameters and a dense scene point cloud. The generated 3D candidates are compared against this point cloud using Normalized Chamfer Distance to select the most geometrically accurate model, filtering out generative hallucinations.
3. Layout Optimization
The system parameterizes each object's translation, rotation, and scale. These parameters are optimized using a dual-loss function: a 3D Chamfer Distance loss (aligning objects to the estimated depth) and a 2D Projection loss (ensuring the 3D object's silhouette matches the original 2D image). This dual-anchor approach prevents common issues like floating objects or incorrect orientations.
Key Results
The proposed method was validated on a custom dataset of 50 multi-object images, including real-world photos, VLM-generated images, and synthetic scenes from the 3D-FRONT benchmark.
Quantitative Performance
Our approach significantly outperforms baselines such as MIDI and Gen3DSR. It achieved the highest CLIP-Scores for both Geometry (0.8389) and Color (0.8990), alongside the lowest 3D Chamfer Distance (0.0127) and a superior 3D F-Score (76.60).
Qualitative and User Evaluation
Qualitatively, the method maintains sharp textures and structural integrity where competitors often produce distorted meshes. In a blind user study with 400 responses, human evaluators preferred our results 55% of the time, ranking it first among all tested approaches.
Conclusion
By combining logical 2D inpainting with joint 3D-2D layout optimization, we achieve state-of-the-art results in single-image 3D scene generation. While the system currently struggles with extreme occlusions (>25% IoU) and complex backgrounds, future work will focus on decoupling foreground assets from environmental lighting and background geometry to enhance photorealism.
Summary updated