Abstract
The production technology of 3D digital content involves multiple stages, including 3D modeling, simulation animation, visualization rendering, and perceptual interaction. It is not only the core technology supporting the creation of 3D digital content but also a key element in enhancing immersive application experiences in virtual reality and the metaverse. A primary focus in computer vision and computer graphics research has been on how to create 3D digital content that is efficient, convenient, controllable, and editable. Currently, producing high-quality 3D digital content still requires significant time and effort from a large number of designers. To address this challenge, leveraging artificial intelligence-generated methods to break down production barriers has emerged as an effective strategy. With the substantial breakthroughs achieved by diffusion models in the field of image generation, they also demonstrate tremendous potential in 3D digital content generation, potentially becoming a foundational model in this area. Recent studies have shown that diffusion model-based techniques for generating 3D digital content can significantly reduce production costs and enhance efficiency. Therefore, it is essential to summarize and categorize existing methods to facilitate further research. This paper systematically reviews 3D digital content generation methods, introducing related 3D representation techniques and focusing on 3D digital content generation schemes, algorithms, and pipeline based on diffusion models. We perform a horizontal comparison of different approaches in terms of generation speed and quality, deeply analyze existing challenges, and propose viable solutions. Furthermore, we thoroughly explore future research themes and directions in this domain, aiming to provide guidance and reference for subsequent research endeavors.
Introduction
Creating high-quality 3D digital content for games, VR, or the metaverse has traditionally been a significant bottleneck in digital production. While AI has revolutionized text and image generation, 3D generation lags behind due to a massive "data gap." For instance, the ImageNet dataset contains 14 million 2D images, whereas standard 3D datasets like ShapeNet contain only about 51,000 objects. Training native 3D generative models from scratch is thus computationally expensive and data-limited.
This paper explores the "lifting" approach: leveraging the vast knowledge embedded in 2D diffusion models—trained on billions of images—to supervise the creation of 3D objects. By using 2D models as "critics," researchers can optimize 3D assets without requiring massive 3D-specific datasets.
Methodology and 3D Representations
The current landscape of 3D generation is categorized into two main tasks: Text-to-3D and Image-to-3D. Several technical pillars support these tasks:
- 3D Representations: Research has shifted from Neural Radiance Fields (NeRF), which use neural networks for volume and color, to 3D Gaussian Splatting (3DGS). 3DGS is increasingly favored for its significantly faster rendering speeds.
- Score Distillation Sampling (SDS): This is the core mechanism for 2D-to-3D lifting. Instead of direct generation, a 3D model renders a 2D view, adds noise, and uses a pre-trained 2D diffusion model to predict that noise. The error in this prediction provides a gradient to update the 3D model's parameters.
- Two-Stage Pipelines: Most successful models utilize a "Coarse-to-Refine" approach. A coarse stage establishes basic geometry, followed by a refinement stage that adds high-resolution textures and fine geometric details.
Performance Analysis
A horizontal comparison conducted on an NVIDIA A40 GPU highlights a critical trade-off between generation speed and quality. In Text-to-3D tasks, ProlificDreamer achieves the highest mesh quality but requires approximately 13 hours. Conversely, DreamGaussian can produce results in just 4 minutes. In Image-to-3D benchmarks using the RealFusion15 dataset, DreamGaussian led in reconstruction accuracy (PSNR of 35.47), while Magic123 excelled in generating novel views (CLIP-Similarity of 0.90) by combining 2D and 3D priors.
Challenges: The Janus Problem
A persistent challenge in this field is the "Janus Problem"—the tendency of models to generate objects with multiple faces (e.g., a dog with two heads). This occurs because the 2D prior lacks a global understanding of 3D consistency; it knows what a face looks like from many angles but doesn't inherently know that an object should only have one front. Additionally, SDS loss often leads to "over-saturated" or unnaturally vibrant textures.
Conclusion and Future Work
While diffusion-based 3D generation has significantly reduced production costs, hurdles in controllability and physical reasoning remain. Future research is expected to integrate Large Language Models (LLMs) for precise editing and develop more "3D-aware" diffusion priors to resolve global consistency issues. Moving from simple visual hallucination to robust 3D reasoning will be the next frontier in automated digital content creation.
Summary updated