TL;DR
This paper tackles the challenge of generating consistent and high-quality 3D models from a single image using diffusion models. They introduce "Unbiased Score Distillation" (USD) to fix a bias in existing multi-view diffusion models, leading to more accurate geometry. Combined with a specialized diffusion model and a novel reference view distillation loss, their method achieves state-of-the-art results without restricting camera poses.
What Problem Does This Paper Address?
Generating 3D assets from a single 2D image is a long-standing goal in computer graphics and vision. Recent advancements in diffusion models have shown promise, but existing methods often struggle with two key issues: 1) inconsistent views (the generated images from different angles don't quite match up) and 2) over-smoothed or inaccurate geometry. Some approaches try to enforce multi-view consistency by adding specialized modules, but this often sacrifices flexibility in camera positioning. This paper aims to achieve both high-quality, consistent multi-view generation and accurate geometry extraction from a single image, while retaining the freedom to specify arbitrary camera poses.
Methods
The core of the paper's contribution lies in two main areas:
- Unbiased Score Distillation (USD): The authors identify a critical bias in the "unconditional noise" predicted by existing multi-view diffusion models like Zero-1-to-3. This bias, stemming from how the model is trained and fine-tuned, leads to inaccuracies when distilling 3D information. USD rectifies this by using unconditional noise from a standard 2D diffusion model (like Stable Diffusion) instead, which significantly improves the fidelity of the optimized radiance field (a representation of the 3D scene).
- Two-Stage Specialization and Reference View Distillation:
- They leverage the optimized radiance field as a "consistency prior."
- They then employ a two-stage "DreamBooth" fine-tuning process on a 2D diffusion model. This specialization makes the diffusion model adept at denoising images of the specific target object. The first stage learns the general visual style, and the second stage learns the object's identity.
- Finally, they introduce a "reference view score distillation" loss during mesh reconstruction (using the NeuS technique). This loss encourages the reconstructed geometry to be consistent with the input image's viewpoint, a crucial aspect often overlooked by other methods.
Datasets
- GSO dataset: Used for quantitative evaluation of geometry reconstruction.
- Google Scanned Object dataset: Also used for evaluation, containing a variety of everyday objects.
- Zero-1-to-3, SyncDreamer, and Wonder3D datasets: Images collected from these sources were used to assess generalization ability.
Key Insights and Ideas
- The "unconditional noise" bias is a critical bottleneck: The paper highlights that the seemingly minor issue of biased unconditional noise in multi-view diffusion models has a significant negative impact on the quality of distilled 3D geometry.
- Leveraging a standard 2D diffuser for unbiased noise is effective: Instead of retraining a complex multi-view model, using the well-behaved unconditional noise from a pre-trained 2D diffusion model is a surprisingly simple yet powerful fix.
- Radiance field as a consistency prior is more rigid than volume/ray aggregation: The authors propose that the optimized radiance field itself can serve as a strong consistency constraint, simplifying the overall pipeline.
- Specializing diffusion models for specific objects is key: Fine-tuning a diffusion model (like with DreamBooth) to a target object dramatically improves its ability to generate high-quality, object-specific views.
- Input view supervision is crucial for geometry: Directly incorporating the input image's viewpoint into the geometry reconstruction loss (via reference view distillation) leads to more faithful and detailed results.
Key Results
- Qualitative: The method generates visually consistent multi-view images and high-quality textured meshes that preserve image details, outperforming baselines like Wonder3D and SyncDreamer, especially for challenging camera poses.
- Quantitative:
- Achieves state-of-the-art results on the GSO dataset for Chamfer Distance and Volume IoU, with their method (Ours) achieving the lowest Chamfer Distance (0.0177) and highest Volume IoU (0.6330) when using iNGP+NeuS.
- In novel view synthesis, they report competitive PSNR, SSIM, and LPIPS scores, with their method achieving 25.38 PSNR, 0.927 SSIM, and 0.049 LPIPS.
- Ablation studies confirm that each component (USD, DreamBooth, reference view distillation) contributes significantly to the overall performance.
Limitations and Future Work
- Bias in Zero-1-to-3: While the paper proposes a fix for the bias in Zero-1-to-3's unconditional noise, the authors acknowledge that the underlying causes of this bias are not fully understood and warrant further theoretical investigation.
- Front-view assumption: The current camera setting assumes a front-view input. For images deviating from this, manual adjustment of polar angles or camera estimation is required.
- Future Work: The authors plan to delve deeper into the theoretical aspects of the bias problem. They also aim to explore the broader applications of their Unbiased Score Distillation (USD) technique in areas like image translation and general view synthesis.
Summary updated