Abstract
3DDesigner is a novel framework designed to extend the success of text-guided 2D diffusion models into the 3D domain. By integrating a NeRF-based coarse prior with a unique two-stream asynchronous diffusion process, the system achieves high-fidelity object generation and consistent 360° local editing. This approach effectively addresses the "Janus problem" and the computational inefficiencies of optimization-heavy 3D generation methods.
Introduction
The rapid advancement of text-to-image diffusion models has set a high bar for photorealism in 2D content creation. However, translating this performance to 3D remains a significant challenge. The primary hurdle is maintaining 3D consistency—ensuring that an object's features remain coherent across all viewing angles. Existing solutions typically rely on slow, per-object optimization or feed-forward models that often suffer from structural hallucinations or lack fine-grained detail.
3DDesigner introduces a multi-stage pipeline that bridges the gap between 2D diffusion and 3D structure, enabling photorealistic generation, one-shot novel view synthesis, and precise local editing.
Methodology
The 3DDesigner framework consists of three primary components designed to balance geometric structure with generative detail.
NeRF-based Condition Module
The process begins with a coarse text prompt and specific camera parameters. A Neural Radiance Field (NeRF) style MLP generates a low-resolution 3D prior. This provides a rough geometric "skeleton" that serves as a spatial condition for the subsequent diffusion stages, ensuring the basic shape of the object is established.
Two-stream Asynchronous Diffusion
To maintain consistency between different views, 3DDesigner employs a novel two-stream asynchronous diffusion module. Unlike standard diffusion processes that denoise images at identical noise levels, this module denoises two views simultaneously at different timesteps ($\Delta t$).
In this "asynchronous" setup, one stream is kept at a lower noise level (cleaner) to serve as a structural anchor for the noisier stream. Features are exchanged between the two U-Net streams via cross-attention mechanisms, forcing the model to learn and enforce correspondences between different perspectives.
3D Local Editing via Noise-to-Text Inversion
Local editing is achieved by modifying a single 2D view and then performing noise-to-text inversion. The system optimizes a specific text embedding so that the diffusion model produces the edited noise pattern. Because the resulting text embedding is view-independent, the edit is naturally propagated across the entire 3D object, maintaining global consistency.
Experimental Results
The model was evaluated using two primary datasets: RealCars, a comprehensive collection of nearly 500,000 images across 11,696 car models with full 360-degree views, and 3D-FUTURE, a standard dataset for industrial furniture models.
Quantitative Performance
3DDesigner demonstrates significant improvements over existing baselines. In quantitative evaluations, the model achieved a Frechet Inception Distance (FID) of 18.77, substantially outperforming CLIP-NeRF (62.97) and 3DiM (35.46). Furthermore, the model showed high controllability, accurately distinguishing between specific vehicle sub-models (e.g., BMW 5 Series vs. 1 Series) with a classification accuracy of 71.33% for generated car types.
Qualitative Consistency
Qualitative analysis confirms that 3DDesigner produces seamless 360-degree rotations. It avoids the "popping" or morphing artifacts common in earlier generative models, providing a stable and photorealistic 3D representation.
Conclusion and Future Work
3DDesigner represents a significant step toward practical, high-quality 3D content generation. By leveraging asynchronous diffusion and noise-to-text inversion, it provides a robust solution for consistent 3D editing and generation.
Despite its success, the framework faces limitations regarding computational cost, as the two-stream architecture is roughly twice as slow as standard diffusion. Future work will focus on expanding the model to open-world generation beyond specific categories and optimizing the training resources required for such large-scale multi-view datasets.
Summary updated