BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion

Yonghao Yu, Shunan Zhu, Huai Qin, Haorui LiIJCAI 2024

Introduction

Generating high-quality 3D assets from text prompts is a cornerstone of modern digital content creation, yet it remains a computationally expensive and technically challenging task. Current methodologies often force a compromise between speed and structural integrity. BoostDream emerges as a "plug-and-play" 3D refining framework designed to bridge this gap, transforming coarse, low-resolution 3D assets into high-fidelity models in approximately 35 minutes while effectively mitigating common artifacts like the "Janus problem."

The Speed vs. Quality Dilemma

The field of text-to-3D generation is currently split between two paradigms:

  1. Feed-forward models (e.g., Shap-E): These are exceptionally fast but produce coarse results due to the limited scale of available 3D training data.
  2. Score Distillation Sampling (SDS) methods (e.g., DreamFusion): These leverage massive 2D image diffusion priors to create high-quality 3D space. However, they are slow and prone to the Janus problem, where the model generates multiple faces or heads because it lacks a global understanding of 3D consistency.

BoostDream combines the rapid initialization of feed-forward models with the high-fidelity refinement of SDS, ensuring both efficiency and geometric accuracy.

The BoostDream Framework

BoostDream utilizes a three-stage pipeline that is compatible with various 3D representations, including NeRF, DMTet, and 3D Gaussian Splatting (3DGS).

1. 3D Representation Initialization

The process begins with a coarse asset generated by a fast model like Shap-E. This asset is distilled into a differentiable 3D representation using a simple $L_1$ loss, establishing a trainable baseline in just 200 iterations.

2. The Boost Stage

This stage introduces a novel Multi-View SDS loss. Rather than optimizing a single camera view, the system renders a 2x2 composite grid of four different perspectives. It employs ControlNet (conditioned on Stable Diffusion 1.5) guided by normal maps extracted from the coarse asset. This ensures the refinement adds detail without deviating from the original's underlying geometry.

The refinement process transforms a coarse input (left) into a high-fidelity asset (center) using normal map guidance (right).

3. The Self-Boost Stage

In the final 3,000 iterations, the model transitions to self-supervision. It uses its own rendered normal maps as guidance to sharpen textures and geometric details, allowing the model to surpass the limitations of the initial coarse input.

Key Insights: Normal Maps and Global Awareness

A critical breakthrough in BoostDream is the use of normal maps as a geometric anchor. Unlike RGB images, normal maps describe surface orientation, providing a blueprint that prevents the diffusion model from hallucinating extra limbs or features.

Normal maps provide a stable geometric blueprint that prevents the hallucination of extra features during the diffusion process.

Furthermore, the 2x2 composite strategy forces "global awareness." By requiring the diffusion model to process four views simultaneously, the system ensures consistency across the front, back, and sides of the object, effectively solving the multi-face issue common in earlier SDS methods.

Results and Performance

BoostDream demonstrates significant improvements in both speed and quality:

  • Efficiency: A BoostDream-NeRF refinement takes approximately 34 minutes on a single V100 GPU, outperforming DreamFusion (~58 minutes) and Magic3D.
  • User Preference: In a study with 20 participants, BoostDream achieved a mean score of 4.23/5, significantly higher than Magic3D (3.28) and DreamFusion (2.62).
  • Robustness: Qualitative evaluations show that BoostDream maintains a single, consistent structure across all views, even for complex prompts where baselines often fail.

Conclusion

BoostDream provides a robust, representation-agnostic solution for high-quality 3D asset generation. While it remains dependent on the initial coarse asset's topology, its ability to rapidly refine geometry and texture while maintaining 3D consistency marks a significant step toward efficient, production-ready text-to-3D tools.

Summary updated

BibTeX
@inproceedings{yu2024boostdream,
  author = {Yonghao Yu and Shunan Zhu and Huai Qin and Haorui Li},
  title = {{BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure