TL;DR
This paper introduces 3DS-Gen, a novel framework for generating consistent 3D scenes from text descriptions. It tackles the challenge of global consistency by first generating a coherent multi-view video and then reconstructing the 3D scene using Gaussian splatting and an adaptive density optimization, offering a practical bridge to 3D content creation.
What Problem Does This Paper Address?
Creating realistic 3D scenes from simple text prompts is a long-standing goal in computer graphics and AI. While recent progress has been made, a major hurdle is maintaining "global consistency" – ensuring that the geometry, appearance, lighting, and layout of the scene remain coherent across all viewpoints. Existing methods often struggle with this, leading to fragmented or inconsistent 3D outputs. Furthermore, many current text-to-3D methods focus on single objects and produce explicit meshes that are difficult to edit natively in standard 3D tools. This paper aims to bridge this gap by generating globally consistent 3D scenes that can be easily reconstructed and, if needed, converted into editable meshes.
Methods
3DS-Gen employs a "generate-then-reconstruct" pipeline, broken down into three key stages:
- Scene Image Sequence Generation: The process begins with a text prompt fed into a Diffusion Transformer (DiT) model. This model, augmented with specific consistency modules (RoPE, AdaIN, QK-Norm), generates a temporally coherent multi-view video sequence. To manage computational costs and temporal redundancy, a cascaded Variational Autoencoder (VAE) is used: a 2D VAE for spatial compression of individual frames and a 3D VAE for temporal compression across the sequence. This results in a compact, coherent latent sequence that serves as a high-quality prior.
- Sparse-view Gaussian Reconstruction: The generated video sequence is then used to estimate camera poses and a sparse 3D geometry scaffold. The paper leverages the Mast3R model, a self-supervised geometry network, for this task, which is adept at estimating accurate camera intrinsics/extrinsics and a near-metric sparse 3D structure. This sparse geometry is then used to initialize 3D Gaussian Splatting (3DGS), a method that represents scenes as a collection of Gaussian ellipsoids. An adaptive density threshold is introduced to intelligently allocate detail within the Gaussian stage, optimizing for quality under a fixed computational budget.
- Mesh Extraction (Optional): While the primary focus is on consistent 3D reconstruction, explicit meshes can be extracted from the optimized 3DGS representation. This is achieved using an enhanced Poisson surface reconstruction technique, which fuses depth and normal cues from the reconstructed scene to generate a watertight surface.
Datasets
The experiments were conducted using publicly available datasets:
- Hypersim: Used for training and validation, offering highly realistic indoor renders with pixel-perfect geometry.
- ScanNet++: A cleaned-up version of ScanNet, used to ensure the system can handle real-world noise.
- SceneNet RGB-D: Used to familiarize the diffusion backbone with motion statistics.
The paper does not introduce new benchmarks but evaluates its performance against existing methods on these established datasets.
Key Insights and Ideas
- Generate-then-Reconstruct for Consistency: The core idea is that generating a coherent video prior first is more effective for achieving global 3D consistency than trying to stitch individual views or build scenes incrementally.
- Cascaded VAE for Efficient Priors: Using a 2D VAE for spatial compression and a 3D VAE for temporal compression provides a compact yet coherent latent representation, balancing fidelity and efficiency.
- Adaptive Density Threshold for 3DGS: Dynamically adjusting the density of Gaussian points during optimization allows for better detail allocation without increasing the overall computational load, leading to improved reconstruction quality.
- Prioritizing Reconstructability over Explicit Mesh Fidelity: The authors emphasize that their primary goal is to produce a consistent and reconstructable 3D representation, from which meshes can be extracted. This acknowledges that direct mesh generation can sometimes sacrifice fine details or introduce artifacts.
Key Results
- Quantitative Comparison: 3DS-Gen achieved the highest PSNR and SSIM scores among tested methods, indicating sharper and more structurally accurate reconstructions. It also yielded the lowest LPIPS values, signifying superior perceptual quality.
- Efficiency: The framework demonstrated significantly faster training times compared to many baseline methods, requiring approximately 30 minutes per scene on a single GPU.
- Qualitative Results: Visualizations showed that 3DS-Gen produced clearer geometry and fewer artifacts than competing methods, maintaining coherent object boundaries even in complex regions.
- Generalization: The system demonstrated strong generalization capabilities, producing coherent and detailed scene models across diverse indoor and outdoor prompts without prompt-specific tuning.
- Ablation Studies: Experiments confirmed the effectiveness of the cascaded VAE design, the choice of Mast3R for sparse reconstruction, and the adaptive density threshold for improving detail fidelity.
Limitations and Future Work
Limitations: * Video Prior Dependence: The ultimate fidelity of the 3D scene is capped by the quality of the initial text-to-video prior. Complex prompts requiring nuanced physical reasoning or fine-grained interactions can still lead to temporal inconsistencies or artifacts. * Reconstruction Challenges: Scenes with large untextured regions, specular/transparent materials, or very thin structures can still pose challenges for reliable feature matching and pose estimation, potentially leading to incomplete geometries. * Mesh Extraction Artifacts: The Poisson reconstruction method, while effective for global structure, can oversmooth high-frequency geometric details and may produce surface roughness or artifacts, especially in normal maps. Producing clean, manifold topologies for complex surfaces remains a challenge.
Future Work: * Stronger Text-to-Video Generators: Integrating more powerful text-to-video models (e.g., CogVideo, Wan) via a lightweight adapter could improve the video prior's semantics and temporal consistency. * Alternative Reconstruction Backends: Exploring NeRF-derived meshers (e.g., NeuS2, 2DGS) as drop-in replacements for GS optimization could potentially yield sharper geometry and cleaner topologies, especially when given more time and views. * Broader Evaluation Metrics: Developing and using metrics that better capture geometry-text alignment and semantic consistency, beyond standard PSNR/SSIM/LPIPS, is suggested.
Summary updated