Introduction
Generating high-quality 3D assets from text prompts is a cornerstone of modern digital content creation, yet it remains a computationally expensive and technically challenging task. Current methodologies often force a compromise between speed and structural integrity. BoostDream emerges as a "plug-and-play" 3D refining framework designed to bridge this gap, transforming coarse, low-resolution 3D assets into high-fidelity models in approximately 35 minutes while effectively mitigating common artifacts like the "Janus problem."
The Speed vs. Quality Dilemma
The field of text-to-3D generation is currently split between two paradigms:
- Feed-forward models (e.g., Shap-E): These are exceptionally fast but produce coarse results due to the limited scale of available 3D training data.
- Score Distillation Sampling (SDS) methods (e.g., DreamFusion): These leverage massive 2D image diffusion priors to create high-quality 3D space. However, they are slow and prone to the Janus problem, where the model generates multiple faces or heads because it lacks a global understanding of 3D consistency.
BoostDream combines the rapid initialization of feed-forward models with the high-fidelity refinement of SDS, ensuring both efficiency and geometric accuracy.
The BoostDream Framework
BoostDream utilizes a three-stage pipeline that is compatible with various 3D representations, including NeRF, DMTet, and 3D Gaussian Splatting (3DGS).
1. 3D Representation Initialization
The process begins with a coarse asset generated by a fast model like Shap-E. This asset is distilled into a differentiable 3D representation using a simple $L_1$ loss, establishing a trainable baseline in just 200 iterations.
2. The Boost Stage
This stage introduces a novel Multi-View SDS loss. Rather than optimizing a single camera view, the system renders a 2x2 composite grid of four different perspectives. It employs ControlNet (conditioned on Stable Diffusion 1.5) guided by normal maps extracted from the coarse asset. This ensures the refinement adds detail without deviating from the original's underlying geometry.
3. The Self-Boost Stage
In the final 3,000 iterations, the model transitions to self-supervision. It uses its own rendered normal maps as guidance to sharpen textures and geometric details, allowing the model to surpass the limitations of the initial coarse input.
Key Insights: Normal Maps and Global Awareness
A critical breakthrough in BoostDream is the use of normal maps as a geometric anchor. Unlike RGB images, normal maps describe surface orientation, providing a blueprint that prevents the diffusion model from hallucinating extra limbs or features.
Furthermore, the 2x2 composite strategy forces "global awareness." By requiring the diffusion model to process four views simultaneously, the system ensures consistency across the front, back, and sides of the object, effectively solving the multi-face issue common in earlier SDS methods.
Results and Performance
BoostDream demonstrates significant improvements in both speed and quality:
- Efficiency: A BoostDream-NeRF refinement takes approximately 34 minutes on a single V100 GPU, outperforming DreamFusion (~58 minutes) and Magic3D.
- User Preference: In a study with 20 participants, BoostDream achieved a mean score of 4.23/5, significantly higher than Magic3D (3.28) and DreamFusion (2.62).
- Robustness: Qualitative evaluations show that BoostDream maintains a single, consistent structure across all views, even for complex prompts where baselines often fail.
Conclusion
BoostDream provides a robust, representation-agnostic solution for high-quality 3D asset generation. While it remains dependent on the initial coarse asset's topology, its ability to rapidly refine geometry and texture while maintaining 3D consistency marks a significant step toward efficient, production-ready text-to-3D tools.
Summary updated