Optimized View and Geometry Distillation from Multi-view Diffuser

Youjia Zhang, Junqing Yu, Zikai Song, Wei Yang

Introduction

Turning a single 2D image into a high-quality 3D object is a "holy grail" of computer vision. Recent methods use Score Distillation Sampling (SDS)—essentially using a 2D diffusion model (like Stable Diffusion) as a "critic" to guide the optimization of a 3D representation, such as a Neural Radiance Field (NeRF). However, current multi-view diffusers (like Zero-1-to-3) often produce blurry, over-smoothed geometry. This paper identifies a fundamental flaw in these models and introduces Unbiased Score Distillation (USD) to rectify it.

The Problem: Noise Bias

The authors discovered that the "unconditional" noise predicted by multi-view models—which is supposed to be a neutral baseline—is actually heavily biased. Because these models weren't trained with proper "classifier-free guidance" (randomly dropping conditions), their baseline noise still "remembers" the object. This leads to mathematical errors during 3D optimization that wash out fine details, resulting in "melted" or blurry surfaces.

Methods

The authors propose a three-pronged approach to achieve state-of-the-art geometry and texture reconstruction:

  1. Unbiased Score Distillation (USD): Instead of using the biased unconditional noise from the multi-view model, they swap it for the "cleaner" unconditional noise from a standard 2D Stable Diffusion model. They also introduce a weighting factor ($\lambda$) to tune the level of detail.
  2. Two-Stage Specified Diffusion: To ensure generated views are perfectly consistent with the original object, they use DreamBooth to fine-tune a 2D model on the specific target object. Stage 1 learns the general "style" from multi-view images, while Stage 2 focuses on "identity" using the original input image.
  3. Geometry and Texture Distillation: They use the optimized NeRF as a "consistency prior." They render views from this NeRF, add noise, and use their specialized DreamBooth model to "clean" them. Finally, NeuS (a neural surface representation) extracts a high-resolution mesh from these refined images.
The proposed two-stage framework for 3D generation, showing the transition from consistency prior to object-specific refinement.

Key Insights

  • The "Bias" Discovery: The failure of multi-view SDS isn't just a lack of data, but a mathematical bias in unconditional branching. Fixing this at the noise level is more efficient than retraining entire models.
  • Radiance Fields as Rigid Priors: Rather than complex inter-view attention modules, a well-optimized NeRF serves as a more reliable "rigid" prior for 3D consistency.
  • Perceptual vs. Pixel Loss: In NeuS reconstruction, using a "View Score Distillation" loss instead of standard Mean Squared Error (MSE) forces the model to prioritize perceptual quality, recovering sharper textures.

Key Results

The USD method outperforms current state-of-the-art (SOTA) models like Wonder3D and SyncDreamer in both geometry and image quality. On the Google Scanned Objects (GSO) dataset, USD achieved a Chamfer Distance of 0.0177 and a Volume IoU of 0.6330, significantly better than Wonder3D (0.0199 / 0.6244).

Qualitative comparison of USD against state-of-the-art methods, demonstrating superior texture sharpness and geometric accuracy.

Beyond metrics, USD allows for arbitrary camera positioning, unlike models restricted to fixed poses. Qualitative results show much sharper textures and more accurate shapes for complex objects, such as figurines and toys.

Conclusion and Future Work

While the current optimization process takes approximately 1.5 hours on an NVIDIA V100, it represents a significant leap in fidelity. The authors suggest that this "unbiased" noise technique could be applied to other fields like image-to-image translation. Future work will explore faster 3D representations, such as 3D Gaussian Splatting, to reduce the computational overhead.

Summary updated

Abstract

Generating multi-view images from a single input view using image-conditioned diffusion models is a recent advancement and has shown considerable potential. However, issues such as the lack of consistency in synthesized views and over-smoothing in extracted geometry persist. Previous methods integrate multi-view consistency modules or impose additional supervisory to enhance view consistency while compromising on the flexibility of camera positioning and limiting the versatility of view synthesis. In this study, we consider the radiance field optimized during geometry extraction as a more rigid consistency prior, compared to volume and ray aggregation used in previous works. We further identify and rectify a critical bias in the traditional radiance field optimization process through score distillation from a multi-view diffuser. We introduce an Unbiased Score Distillation (USD) that utilizes unconditioned noises from a 2D diffusion model, greatly refining the radiance field fidelity. We leverage the rendered views from the optimized radiance field as the basis and develop a two-step specialization process of a 2D diffusion model, which is adept at conducting object-specific denoising and generating high-quality multi-view images. Finally, we recover faithful geometry and texture directly from the refined multi-view images. Empirical evaluations demonstrate that our optimized geometry and view distillation technique generates comparable results to the state-of-the-art models trained on extensive datasets, all while maintaining freedom in camera positioning. Please see our project page at https://youjiazhang.github.io/USD/.
BibTeX
@inproceedings{zhang2025optimized,
  author = {Youjia Zhang and Junqing Yu and Zikai Song and Wei Yang},
  title = {{Optimized View and Geometry Distillation from Multi-view Diffuser}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/269},
}
Figure