Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization

Xiang Tang, Ruotong Li, Xiaopeng Fan

Abstract

Generating 3D scenes from a single RGB image remains a significant challenge due to object occlusion and depth ambiguity. We propose a novel three-stage framework that ensures both high-quality object generation and scene coherence. Our method begins with instance segmentation and inpainting to recover occluded details, followed by 3D asset generation. We then estimate scene geometry via pseudo-stereo viewpoints to select the best-matching models. Finally, we optimize the spatial layout by minimizing Chamfer distance in both 3D and 2D projection spaces. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in geometric accuracy, texture fidelity, and layout synthesis.

Introduction

While AI has made significant progress in generating single 3D objects, creating entire multi-object scenes from a single viewpoint remains difficult. Current methods often produce "melted" geometry or fail to place objects correctly in 3D space due to flawed monocular depth estimates. This paper introduces a "divide-and-conquer" framework that decouples object generation from layout optimization to maintain high-fidelity details while ensuring spatial coherence.

Methodology

Our framework follows a three-stage "decomposition-recomposition" pipeline designed to handle the complexities of multi-object scenes.

The three-stage framework: Point cloud extraction, Instance Segmentation and Generation, and Layout Optimization.

1. Instance Segmentation and Generation

The system first detects and segments individual objects using Grounding DINO and SAM. To handle occlusions, a Vision-Language Model (GPT-4o) inpaints missing parts of the 2D segments. These complete images are then used by a 3D generator (Trellis) to create candidate meshes for each object.

2. Point Cloud Extraction & Model Selection

Using DUSt3R, the framework creates a pseudo-stereo pair from the input image to estimate camera parameters and a dense scene point cloud. The generated 3D candidates are compared against this point cloud using Normalized Chamfer Distance to select the most geometrically accurate model, filtering out generative hallucinations.

3. Layout Optimization

The system parameterizes each object's translation, rotation, and scale. These parameters are optimized using a dual-loss function: a 3D Chamfer Distance loss (aligning objects to the estimated depth) and a 2D Projection loss (ensuring the 3D object's silhouette matches the original 2D image). This dual-anchor approach prevents common issues like floating objects or incorrect orientations.

Key Results

The proposed method was validated on a custom dataset of 50 multi-object images, including real-world photos, VLM-generated images, and synthetic scenes from the 3D-FRONT benchmark.

Qualitative results showing the transition from input images to scene geometry and final textured 3D scenes across various indoor and tabletop environments.

Quantitative Performance

Our approach significantly outperforms baselines such as MIDI and Gen3DSR. It achieved the highest CLIP-Scores for both Geometry (0.8389) and Color (0.8990), alongside the lowest 3D Chamfer Distance (0.0127) and a superior 3D F-Score (76.60).

Qualitative and User Evaluation

Qualitatively, the method maintains sharp textures and structural integrity where competitors often produce distorted meshes. In a blind user study with 400 responses, human evaluators preferred our results 55% of the time, ranking it first among all tested approaches.

Conclusion

By combining logical 2D inpainting with joint 3D-2D layout optimization, we achieve state-of-the-art results in single-image 3D scene generation. While the system currently struggles with extreme occlusions (>25% IoU) and complex backgrounds, future work will focus on decoupling foreground assets from environmental lighting and background geometry to enhance photorealism.

Summary updated

Abstract

In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
BibTeX
@misc{tang2025towards,
  author = {Xiang Tang and Ruotong Li and Xiaopeng Fan},
  title = {{Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization}},
  year = {2025},
  eprint = {2507.14841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.14841},
}
Figure