TL;DR
This paper introduces Unbiased Score Distillation (USD), a technique that rectifies inherent biases in multi-view diffusion models to enable high-fidelity 3D reconstruction from a single image. By combining a rigid radiance field consistency prior with a two-stage object-specialized denoising process, the authors achieve state-of-the-art geometry and texture quality without sacrificing the flexibility of camera positioning.
What Problem Does This Paper Address?
Generating a complete 3D object from a single 2D image is a "holy grail" task in computer vision. While recent image-conditioned diffusion models like Zero-1-to-3 have made strides, they suffer from two major flaws:
- View Inconsistency: Different synthesized viewpoints of the same object often don't "line up" geometrically.
- Over-smoothing: When these models are used to "distill" 3D shapes via Score Distillation Sampling (SDS), the resulting geometry often lacks fine detail and looks blurry.
Existing solutions often try to force consistency by adding specialized modules or restricting the camera to a few fixed poses. However, these methods limit the versatility of the synthesis and often struggle with "in-the-wild" images captured from unusual angles.
Methods
The authors propose a multi-stage pipeline that treats a Neural Radiance Field (NeRF) as a rigid consistency prior.
1. Unbiased Score Distillation (USD)
The authors discovered that the "unconditional noise" predicted by models like Zero-1-to-3 is fundamentally biased. Because these models weren't trained with proper classifier-free guidance (CFG) protocols, using their noise predictions for 3D optimization leads to distorted results.
To fix this, USD swaps the biased unconditional noise of the multi-view diffuser with noise from a standard, pre-trained 2D model (Stable Diffusion). This simple "rectification" significantly improves the fidelity of the initial 3D radiance field.
2. Two-Stage Specialization (DreamBooth)
To capture object-specific details, the authors fine-tune a 2D diffusion model using the DreamBooth technique in two stages:
- Stage 1: The model learns the general visual style using multi-view images generated by Zero-1-to-3.
- Stage 2: The model focuses on fine details by treating the original input image as the primary "ground truth" reference.
3. Geometry and Texture Refinement
Finally, the system renders views from the coarse NeRF, adds a small amount of noise, and uses the specialized DreamBooth model to denoise them into high-quality, consistent images. These refined images are then used to reconstruct the final 3D mesh using the NeuS technique, supported by a novel Reference View Distillation Loss ($L_{RV}$) that ensures the final 3D model remains perceptually faithful to the original input image.
Datasets
The authors evaluated their method using:
- Google Scanned Objects (GSO): A dataset of 30 common household items and animals used for quantitative benchmarking.
- RTMV & Objaverse: Used for empirical validation of the noise rectification.
- In-the-wild images: Collected from various project pages to test generalization.
Key Insights and Ideas
- The Bias Discovery: The realization that multi-view diffusers fail at 3D distillation not because of the 3D task itself, but because of a training flaw in how they handle unconditional noise.
- NeRF as a Rigid Prior: Instead of complex attention mechanisms to keep views consistent, the authors show that a well-optimized radiance field is a more effective "anchor" for consistency.
- Perceptual vs. Pixel Loss: The use of score distillation ($L_{RV}$) at the input viewpoint is superior to standard Mean Squared Error (MSE) because it focuses on the perceptual quality and structure rather than just matching raw pixel values, which often leads to artifacts.
Key Results
The method produces 3D reconstructions that are sharper and more geometrically accurate than previous state-of-the-art (SOTA) methods like SyncDreamer and Wonder3D.
- Quantitative: On the GSO dataset, the method achieved a Chamfer Distance of 0.0177 and a Volume IoU of 0.6330, outperforming Wonder3D (0.0199 / 0.6244).
- Qualitative: The system excels at maintaining structure even when the input image is taken from a high or unusual viewing angle—a scenario where other models typically fail.
Limitations and Future Work
- Computational Cost: The full pipeline, including NeRF optimization and DreamBooth fine-tuning, takes approximately 1.5 hours on an NVIDIA V100 GPU. This is significantly slower than "feed-forward" methods that generate 3D in seconds.
- Theoretical Underpinnings: While the authors identified and fixed the noise bias, the deep theoretical reasons why this bias emerges during the training of multi-view diffusers remain an open question for future research.
- Future Directions: The authors suggest exploring USD in other domains, such as general image-to-image translation and more complex view synthesis tasks.
Summary updated