Optimized View and Geometry Distillation from Multi-view Diffuser

Youjia Zhang, Zikai Song, Junqing Yu, Yawei Luo, Wei Yang

TL;DR

This paper tackles the challenge of generating consistent and high-quality 3D models from a single image using diffusion models. They introduce "Unbiased Score Distillation" (USD) to fix a bias in existing multi-view diffusion models, leading to more accurate geometry. Combined with a specialized diffusion model and a novel reference view distillation loss, their method achieves state-of-the-art results without restricting camera poses.

What Problem Does This Paper Address?

Generating 3D assets from a single 2D image is a long-standing goal in computer graphics and vision. Recent advancements in diffusion models have shown promise, but existing methods often struggle with two key issues: 1) inconsistent views (the generated images from different angles don't quite match up) and 2) over-smoothed or inaccurate geometry. Some approaches try to enforce multi-view consistency by adding specialized modules, but this often sacrifices flexibility in camera positioning. This paper aims to achieve both high-quality, consistent multi-view generation and accurate geometry extraction from a single image, while retaining the freedom to specify arbitrary camera poses.

Methods

The core of the paper's contribution lies in two main areas:

  • Unbiased Score Distillation (USD): The authors identify a critical bias in the "unconditional noise" predicted by existing multi-view diffusion models like Zero-1-to-3. This bias, stemming from how the model is trained and fine-tuned, leads to inaccuracies when distilling 3D information. USD rectifies this by using unconditional noise from a standard 2D diffusion model (like Stable Diffusion) instead, which significantly improves the fidelity of the optimized radiance field (a representation of the 3D scene).
  • Two-Stage Specialization and Reference View Distillation:
    • They leverage the optimized radiance field as a "consistency prior."
    • They then employ a two-stage "DreamBooth" fine-tuning process on a 2D diffusion model. This specialization makes the diffusion model adept at denoising images of the specific target object. The first stage learns the general visual style, and the second stage learns the object's identity.
  • Finally, they introduce a "reference view score distillation" loss during mesh reconstruction (using the NeuS technique). This loss encourages the reconstructed geometry to be consistent with the input image's viewpoint, a crucial aspect often overlooked by other methods.

Datasets

  • GSO dataset: Used for quantitative evaluation of geometry reconstruction.
  • Google Scanned Object dataset: Also used for evaluation, containing a variety of everyday objects.
  • Zero-1-to-3, SyncDreamer, and Wonder3D datasets: Images collected from these sources were used to assess generalization ability.

Key Insights and Ideas

  • The "unconditional noise" bias is a critical bottleneck: The paper highlights that the seemingly minor issue of biased unconditional noise in multi-view diffusion models has a significant negative impact on the quality of distilled 3D geometry.
  • Leveraging a standard 2D diffuser for unbiased noise is effective: Instead of retraining a complex multi-view model, using the well-behaved unconditional noise from a pre-trained 2D diffusion model is a surprisingly simple yet powerful fix.
  • Radiance field as a consistency prior is more rigid than volume/ray aggregation: The authors propose that the optimized radiance field itself can serve as a strong consistency constraint, simplifying the overall pipeline.
  • Specializing diffusion models for specific objects is key: Fine-tuning a diffusion model (like with DreamBooth) to a target object dramatically improves its ability to generate high-quality, object-specific views.
  • Input view supervision is crucial for geometry: Directly incorporating the input image's viewpoint into the geometry reconstruction loss (via reference view distillation) leads to more faithful and detailed results.

Key Results

  • Qualitative: The method generates visually consistent multi-view images and high-quality textured meshes that preserve image details, outperforming baselines like Wonder3D and SyncDreamer, especially for challenging camera poses.
  • Quantitative:
    • Achieves state-of-the-art results on the GSO dataset for Chamfer Distance and Volume IoU, with their method (Ours) achieving the lowest Chamfer Distance (0.0177) and highest Volume IoU (0.6330) when using iNGP+NeuS.
  • In novel view synthesis, they report competitive PSNR, SSIM, and LPIPS scores, with their method achieving 25.38 PSNR, 0.927 SSIM, and 0.049 LPIPS.
  • Ablation studies confirm that each component (USD, DreamBooth, reference view distillation) contributes significantly to the overall performance.

Limitations and Future Work

  • Bias in Zero-1-to-3: While the paper proposes a fix for the bias in Zero-1-to-3's unconditional noise, the authors acknowledge that the underlying causes of this bias are not fully understood and warrant further theoretical investigation.
  • Front-view assumption: The current camera setting assumes a front-view input. For images deviating from this, manual adjustment of polar angles or camera estimation is required.
  • Future Work: The authors plan to delve deeper into the theoretical aspects of the bias problem. They also aim to explore the broader applications of their Unbiased Score Distillation (USD) technique in areas like image translation and general view synthesis.

Summary updated

Abstract

Generating multi-view images from a single input view using image-conditioned diffusion models is a recent advancement and has shown considerable potential. However, issues such as the lack of consistency in synthesized views and over-smoothing in extracted geometry persist. Previous methods integrate multi-view consistency modules or impose additional supervisory to enhance view consistency while compromising on the flexibility of camera positioning and limiting the versatility of view synthesis. In this study, we consider the radiance field optimized during geometry extraction as a more rigid consistency prior, compared to volume and ray aggregation used in previous works. We further identify and rectify a critical bias in the traditional radiance field optimization process through score distillation from a multi-view diffuser. We introduce an Unbiased Score Distillation (USD) that utilizes unconditioned noises from a 2D diffusion model, greatly refining the radiance field fidelity. We leverage the rendered views from the optimized radiance field as the basis and develop a two-step specialization process of a 2D diffusion model, which is adept at conducting object-specific denoising and generating high-quality multi-view images. Finally, we recover faithful geometry and texture directly from the refined multi-view images. Empirical evaluations demonstrate that our optimized geometry and view distillation technique generates comparable results to the state-of-the-art models trained on extensive datasets, all while maintaining freedom in camera positioning. Source code of our work is publicly available at: https://youjiazhang.github.io/USD/.
BibTeX
@inproceedings{zhang2025optimized,
  author = {Youjia Zhang and Zikai Song and Junqing Yu and Yawei Luo and Wei Yang},
  title = {{Optimized View and Geometry Distillation from Multi-view Diffuser}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/269},
}
Figure