The central challenge in 3D computer vision is a massive data imbalance: while we have billions of captioned 2D images to train generative models, high-quality 3D assets are scarce and difficult to curate. To bridge this gap, researchers have developed a way to "lift" the knowledge trapped in 2D diffusion models into the third dimension.
Instead of training a 3D model from scratch, this approach treats a pre-trained 2D diffusion model as a "teacher" or "critic" that guides the optimization of a 3D representation.
The 2D Critic and the 3D Student
The breakthrough that enabled high-fidelity text-to-3D without 3D supervision is Score Distillation Sampling (SDS), introduced by Ben Poole in the seminal DreamFusion paper.
The intuition behind SDS is to treat 3D generation as an optimization problem. We start with a 3D representation—typically a Neural Radiance Field (NeRF) or 3D Gaussian Splatting—and a text prompt like "a DSLR photo of a peacock." To make the 3D model match the text, we follow a four-step loop:
- Render: Create a 2D image of the 3D model from a random camera angle.
- Corrupt: Add a specific amount of random noise to that 2D image.
- Consult the Critic: Pass the noisy image and the text prompt to a frozen 2D diffusion model (like Stable Diffusion). The diffusion model predicts how to "denoise" the image to make it look more like a real peacock.
- Update: The difference between the added noise and the predicted noise provides a gradient. We backpropagate this gradient through the differentiable renderer to update the 3D model’s geometry and colors.
By repeating this thousands of times from different angles, the 3D model is "sculpted" until every possible 2D projection looks like a high-probability image according to the diffusion model. As noted in surveys by Chaoning Zhang and Chenhan Jiang, this mechanism allows researchers to bypass the need for 3D training data entirely.
Refining the Gradient
While the original SDS was revolutionary, it often produced over-saturated, "cartoonish" results or suffered from the Janus problem—where an object (like a dog) might end up with multiple faces because the 2D critic only sees one view at a time and doesn't realize it already "drew" a face on the other side.
To solve these issues, researchers have proposed several specialized distillation variants:
- Classifier Score Distillation (CSD): Xin Yu and colleagues argued in Text-to-3D with Classifier Score Distillation that the "generative" part of the diffusion model can actually introduce noise. By isolating the implicit classifier within the model, CSD focuses purely on aligning the asset with the text prompt, leading to sharper details and better color.
- Self Score Distillation (SSD): In HeadArtist, Yujun Shen and Ziyu Wan used SSD to eliminate multi-face artifacts in 3D heads. By comparing two different noise predictions from the same model, the system can "self-correct" its geometric consistency.
- Reference-Guided SDS (Ref-SDS): When a text prompt isn't enough, Jingbo Zhang and Xiaoyu Li showed in HumanRef that injecting features from a single 2D reference image into the distillation process helps maintain photorealistic textures on the "hidden" sides of a 3D human model.
Beyond Static Objects
The power of score distillation extends beyond just creating static statues. Because the renderer is differentiable, the distillation gradient can be applied to almost any parameterized 3D representation:
- Human Avatars: Ruixiang Jiang and Menglei Chai developed AvatarCraft, which applies SDS to SMPL (a parametric human body model). This ensures the resulting 3D person is not just a static shape but is immediately ready for animation.
- PBR Textures: Instead of optimizing a cloud of points, Paint-it by Kim Youwang uses SDS to optimize physically-based rendering (PBR) maps. This produces professional-grade assets that react realistically to lighting.
- 3D Editing: Methods like GSEdit and RePaint-NeRF use distillation to modify existing 3D scenes. By masking a specific area and applying SDS gradients based on an "editing" prompt (e.g., "make this chair look like it's made of ice"), users can perform localized 3D transformations.
The Future of the Pipeline
As the field matures, the focus is shifting from pure quality to speed and control. OrientDream introduces explicit camera orientation control to the diffusion model to further mitigate the Janus problem, while C3V uses Large Language Models as "directors" to orchestrate multiple SDS-generated assets into complex, 3D-aware videos.
By treating 2D models as a universal source of visual "truth," score distillation has turned the problem of 3D generation into a problem of effective communication between a 2D teacher and a 3D student.