The core challenge in 3D content creation is the lack of massive, labeled 3D datasets. To bypass this, researchers have developed a technique to "lift" the knowledge from 2D image generators into 3D space. By using a pre-trained 2D diffusion model as a critic, we can optimize a 3D representation—like a Neural Radiance Field (NeRF) or 3D Gaussian Splatting—until its 2D renders look exactly like what the diffusion model expects to see.
The Critic in the Loop
The foundational mechanism for this research direction is Score Distillation Sampling (SDS), introduced by Ben Poole, Ajay Jain, and colleagues in DreamFusion: Text-to-3D using 2D Diffusion.
Instead of training a 3D model directly, SDS treats a 2D diffusion model as a frozen "expert." During optimization, the 3D scene is rendered from a random viewpoint. We add noise to this render and ask the diffusion model to predict the noise. The difference between the added noise and the predicted noise provides a gradient that tells us how to update the 3D scene's geometry and texture to make it more "realistic" according to the 2D prior.
Overcoming the Smoothing Problem
While SDS enabled 3D generation without 3D data, it introduced a persistent set of artifacts: over-smoothing and over-saturation. Because the optimization objective tends to seek the "mode" (the most average, high-probability version) of the diffusion distribution, the resulting assets often look like waxy, plastic toys rather than detailed objects.
Recent advancements focus on refining this distillation objective to recover high-frequency details:
- Trajectory Matching: Yixun Liang and others proposed Interval Score Matching (ISM) in LucidDreamer, which uses deterministic diffusion trajectories to provide more precise guidance than the stochastic nature of SDS.
- Noise Removal: Oren Katzir and Daniel Cohen-Or identified a specific noise term in the original SDS formula that causes artifacts. Their method, Noise-free Score Distillation (NFSD), removes this term to achieve sharper textures without extra cost.
- Consistency and Balance: Works like Connecting Consistency Distillation to Score Distillation by Minghui Hu frame the problem as maintaining "cross-consistency" between different views, while Target-Balanced Score Distillation uses negative prompts to prevent the model from collapsing into over-saturated colors.
From Optimization to Generation
The field is shifting from slow, per-scene optimization toward faster, more robust frameworks. CAD: Photorealistic 3D Generation via Adversarial Distillation by Despoina Paschalidou and colleagues moves away from pure score distillation toward adversarial training. By using a 3D-aware GAN to model the distribution of the diffusion prior, they can generate diverse 3D objects in under a second, effectively solving the "Janus-face" problem where objects appear to have multiple fronts.
By treating 2D diffusion models as a rich source of geometric and textural priors, these methods have turned the "data scarcity" problem of 3D deep learning into an optimization problem, steadily closing the gap between AI-generated images and fully realized 3D assets.