Optimized View and Geometry Distillation from Multi-view Diffuser

Youjia Zhang, Zikai Song, Junqing Yu, Yawei Luo, Wei Yang

TL;DR

This paper introduces Unbiased Score Distillation (USD), a technique that rectifies inherent biases in multi-view diffusion models to enable high-fidelity 3D reconstruction from a single image. By combining a rigid radiance field consistency prior with a two-stage object-specialized denoising process, the authors achieve state-of-the-art geometry and texture quality without sacrificing the flexibility of camera positioning.

What Problem Does This Paper Address?

Generating a complete 3D object from a single 2D image is a "holy grail" task in computer vision. While recent image-conditioned diffusion models like Zero-1-to-3 have made strides, they suffer from two major flaws:

  1. View Inconsistency: Different synthesized viewpoints of the same object often don't "line up" geometrically.
  2. Over-smoothing: When these models are used to "distill" 3D shapes via Score Distillation Sampling (SDS), the resulting geometry often lacks fine detail and looks blurry.

Existing solutions often try to force consistency by adding specialized modules or restricting the camera to a few fixed poses. However, these methods limit the versatility of the synthesis and often struggle with "in-the-wild" images captured from unusual angles.

Methods

The authors propose a multi-stage pipeline that treats a Neural Radiance Field (NeRF) as a rigid consistency prior.

1. Unbiased Score Distillation (USD)

The authors discovered that the "unconditional noise" predicted by models like Zero-1-to-3 is fundamentally biased. Because these models weren't trained with proper classifier-free guidance (CFG) protocols, using their noise predictions for 3D optimization leads to distorted results.

Figure 1: The unconditional noise predicted by Zero-1-to-3 model tends to be biased. As a demonstration, we use the ‘Mario’ image as a toy example and add various levels of noise to the image (larger T means more noise has been added). We use the predicted uncondi- tional noise to recover the original image from noisy input and find the results of Zero-1-to-3 deviate from the input image greatly even for very small amount of noise. The right sub-figure shows the av- eraged difference between the predicted noise and the added noise.

To fix this, USD swaps the biased unconditional noise of the multi-view diffuser with noise from a standard, pre-trained 2D model (Stable Diffusion). This simple "rectification" significantly improves the fidelity of the initial 3D radiance field.

2. Two-Stage Specialization (DreamBooth)

To capture object-specific details, the authors fine-tune a 2D diffusion model using the DreamBooth technique in two stages:

  • Stage 1: The model learns the general visual style using multi-view images generated by Zero-1-to-3.
  • Stage 2: The model focuses on fine details by treating the original input image as the primary "ground truth" reference.

3. Geometry and Texture Refinement

Finally, the system renders views from the coarse NeRF, adds a small amount of noise, and uses the specialized DreamBooth model to denoise them into high-quality, consistent images. These refined images are then used to reconstruct the final 3D mesh using the NeuS technique, supported by a novel Reference View Distillation Loss ($L_{RV}$) that ensures the final 3D model remains perceptually faithful to the original input image.

Figure 2: The overall pipeline of our approach. We first use our Unbiased Score Distillation to extract an optimized underlying radiance field. And then we use the NeRF as our consistency prior, i.e., the generated views should be consistent with the NeRF renderings. We propose a two-stage specialization scheme to obtain a specified DreamBooth specifically for the target. We then denoise the NeRF renderings to obtain high-quality views and subsequently use NeuS technique to recover the geometry. Our optimized scheme generates comparable, sometimes better particularly for irregular camera poses, results to the SOTA works without training on large-scale data.

Datasets

The authors evaluated their method using:

  • Google Scanned Objects (GSO): A dataset of 30 common household items and animals used for quantitative benchmarking.
  • RTMV & Objaverse: Used for empirical validation of the noise rectification.
  • In-the-wild images: Collected from various project pages to test generalization.

Key Insights and Ideas

  • The Bias Discovery: The realization that multi-view diffusers fail at 3D distillation not because of the 3D task itself, but because of a training flaw in how they handle unconditional noise.
  • NeRF as a Rigid Prior: Instead of complex attention mechanisms to keep views consistent, the authors show that a well-optimized radiance field is a more effective "anchor" for consistency.
  • Perceptual vs. Pixel Loss: The use of score distillation ($L_{RV}$) at the input viewpoint is superior to standard Mean Squared Error (MSE) because it focuses on the perceptual quality and structure rather than just matching raw pixel values, which often leads to artifacts.

Key Results

The method produces 3D reconstructions that are sharper and more geometrically accurate than previous state-of-the-art (SOTA) methods like SyncDreamer and Wonder3D.

  • Quantitative: On the GSO dataset, the method achieved a Chamfer Distance of 0.0177 and a Volume IoU of 0.6330, outperforming Wonder3D (0.0199 / 0.6244).
  • Qualitative: The system excels at maintaining structure even when the input image is taken from a high or unusual viewing angle—a scenario where other models typically fail.
Figure 5: Qualitative comparisons of our method with baseline approaches, namely Wonder3D, SyncDreamer, and One-2-3-45, on the GSO dataset, focusing on the quality of the reconstructed textured meshes.

Limitations and Future Work

  • Computational Cost: The full pipeline, including NeRF optimization and DreamBooth fine-tuning, takes approximately 1.5 hours on an NVIDIA V100 GPU. This is significantly slower than "feed-forward" methods that generate 3D in seconds.
  • Theoretical Underpinnings: While the authors identified and fixed the noise bias, the deep theoretical reasons why this bias emerges during the training of multi-view diffusers remain an open question for future research.
  • Future Directions: The authors suggest exploring USD in other domains, such as general image-to-image translation and more complex view synthesis tasks.

Summary updated

Abstract

Generating multi-view images from a single input view using image-conditioned diffusion models is a recent advancement and has shown considerable potential. However, issues such as the lack of consistency in synthesized views and over-smoothing in extracted geometry persist. Previous methods integrate multi-view consistency modules or impose additional supervisory to enhance view consistency while compromising on the flexibility of camera positioning and limiting the versatility of view synthesis. In this study, we consider the radiance field optimized during geometry extraction as a more rigid consistency prior, compared to volume and ray aggregation used in previous works. We further identify and rectify a critical bias in the traditional radiance field optimization process through score distillation from a multi-view diffuser. We introduce an Unbiased Score Distillation (USD) that utilizes unconditioned noises from a 2D diffusion model, greatly refining the radiance field fidelity. We leverage the rendered views from the optimized radiance field as the basis and develop a two-step specialization process of a 2D diffusion model, which is adept at conducting object-specific denoising and generating high-quality multi-view images. Finally, we recover faithful geometry and texture directly from the refined multi-view images. Empirical evaluations demonstrate that our optimized geometry and view distillation technique generates comparable results to the state-of-the-art models trained on extensive datasets, all while maintaining freedom in camera positioning. Please see our project page at https://youjiazhang.github.io/USD/.
BibTeX
@inproceedings{zhang2025optimized,
  author = {Youjia Zhang and Zikai Song and Junqing Yu and Yawei Luo and Wei Yang},
  title = {{Optimized View and Geometry Distillation from Multi-view Diffuser}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/269},
}
Figure