3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models

Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, Dacheng Tao

Abstract

3DDesigner is a novel framework designed to extend the success of text-guided 2D diffusion models into the 3D domain. By integrating a NeRF-based coarse prior with a unique two-stream asynchronous diffusion process, the system achieves high-fidelity object generation and consistent 360° local editing. This approach effectively addresses the "Janus problem" and the computational inefficiencies of optimization-heavy 3D generation methods.

Introduction

The rapid advancement of text-to-image diffusion models has set a high bar for photorealism in 2D content creation. However, translating this performance to 3D remains a significant challenge. The primary hurdle is maintaining 3D consistency—ensuring that an object's features remain coherent across all viewing angles. Existing solutions typically rely on slow, per-object optimization or feed-forward models that often suffer from structural hallucinations or lack fine-grained detail.

3DDesigner introduces a multi-stage pipeline that bridges the gap between 2D diffusion and 3D structure, enabling photorealistic generation, one-shot novel view synthesis, and precise local editing.

Methodology

The 3DDesigner framework consists of three primary components designed to balance geometric structure with generative detail.

NeRF-based Condition Module

The process begins with a coarse text prompt and specific camera parameters. A Neural Radiance Field (NeRF) style MLP generates a low-resolution 3D prior. This provides a rough geometric "skeleton" that serves as a spatial condition for the subsequent diffusion stages, ensuring the basic shape of the object is established.

Two-stream Asynchronous Diffusion

To maintain consistency between different views, 3DDesigner employs a novel two-stream asynchronous diffusion module. Unlike standard diffusion processes that denoise images at identical noise levels, this module denoises two views simultaneously at different timesteps ($\Delta t$).

The 3DDesigner architecture featuring the NeRF-based condition module and the two-stream asynchronous diffusion module.

In this "asynchronous" setup, one stream is kept at a lower noise level (cleaner) to serve as a structural anchor for the noisier stream. Features are exchanged between the two U-Net streams via cross-attention mechanisms, forcing the model to learn and enforce correspondences between different perspectives.

3D Local Editing via Noise-to-Text Inversion

Local editing is achieved by modifying a single 2D view and then performing noise-to-text inversion. The system optimizes a specific text embedding so that the diffusion model produces the edited noise pattern. Because the resulting text embedding is view-independent, the edit is naturally propagated across the entire 3D object, maintaining global consistency.

Experimental Results

The model was evaluated using two primary datasets: RealCars, a comprehensive collection of nearly 500,000 images across 11,696 car models with full 360-degree views, and 3D-FUTURE, a standard dataset for industrial furniture models.

Examples of controllable 3D generation and local editing using 3DDesigner.

Quantitative Performance

3DDesigner demonstrates significant improvements over existing baselines. In quantitative evaluations, the model achieved a Frechet Inception Distance (FID) of 18.77, substantially outperforming CLIP-NeRF (62.97) and 3DiM (35.46). Furthermore, the model showed high controllability, accurately distinguishing between specific vehicle sub-models (e.g., BMW 5 Series vs. 1 Series) with a classification accuracy of 71.33% for generated car types.

Qualitative Consistency

Qualitative analysis confirms that 3DDesigner produces seamless 360-degree rotations. It avoids the "popping" or morphing artifacts common in earlier generative models, providing a stable and photorealistic 3D representation.

Conclusion and Future Work

3DDesigner represents a significant step toward practical, high-quality 3D content generation. By leveraging asynchronous diffusion and noise-to-text inversion, it provides a robust solution for consistent 3D editing and generation.

Despite its success, the framework faces limitations regarding computational cost, as the two-stream architecture is roughly twice as slow as standard diffusion. Future work will focus on expanding the model to open-world generation beyond specific categories and optimizing the training resources required for such large-scale multi-view datasets.

Summary updated

Abstract

Text-guided diffusion models have shown superior performance in image/video generation and editing. While few explorations have been performed in 3D scenarios. In this paper, we discuss three fundamental and interesting problems on this topic. First, we equip text-guided diffusion models to achieve 3D-consistent generation. Specifically, we integrate a NeRF-like neural field to generate low-resolution coarse results for a given camera view. Such results can provide 3D priors as condition information for the following diffusion process. During denoising diffusion, we further enhance the 3D consistency by modeling cross-view correspondences with a novel two-stream (corresponding to two different views) asynchronous diffusion process. Second, we study 3D local editing and propose a two-step solution that can generate 360-degree manipulated results by editing an object from a single view. Step 1, we propose to perform 2D local editing by blending the predicted noises. Step 2, we conduct a noise-to-text inversion process that maps 2D blended noises into the view-independent text embedding space. Once the corresponding text embedding is obtained, 360-degree images can be generated. Last but not least, we extend our model to perform one-shot novel view synthesis by fine-tuning on a single image, firstly showing the potential of leveraging text guidance for novel view synthesis. Extensive experiments and various applications show the prowess of our 3DDesigner. The project page is available at https://3ddesigner-diffusion.github.io/.
BibTeX
@misc{li2022ddesigner,
  author = {Gang Li and Heliang Zheng and Chaoyue Wang and Chang Li and Changwen Zheng and Dacheng Tao},
  title = {{3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models}},
  year = {2022},
  eprint = {2211.14108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.14108},
}
Figure