High Fidelity 3D Generation via Improved Score Distillation

This direction is merged into another direction: score distillation artifacts. Its last version stays readable and citable here.

Version 1 of 146 papers47 conceptsfrozen 2026-10-04

The breakthrough in text-to-3D generation arrived when researchers realized they didn't need a 3D dataset to create 3D objects. Instead, they could use a pre-trained 2D diffusion model as a "critic." By rendering a 3D model from a random viewpoint and asking the 2D model how to "improve" the image, we can backpropagate those suggestions into the 3D representation. This process, known as Score Distillation Sampling (SDS), acts as a bridge between the vast knowledge of 2D image distributions and the spatial requirements of 3D assets.

However, this bridge is notoriously rickety. Standard SDS often produces "deep-fried" results: images with extreme over-saturation, a lack of fine detail, and the "Janus problem," where objects (like a person or a dog) grow multiple faces to satisfy the 2D critic from every angle. Recent research has focused on stabilizing this bridge, reinterpreting its mathematical foundations, and introducing structural anchors to ensure the 3D output matches the quality of the 2D prior.

The Over-Optimization Trap

The primary reason SDS fails to produce photorealistic results is a mismatch in how we use the diffusion model. In 2D generation, we sample from the model; in 3D generation, we optimize against it.

To ensure the 3D model converges, researchers often use extremely high Classifier-Free Guidance (CFG) scales. As Xiaofeng Yang and colleagues note in LODS, this pushes the optimization toward the "modes" of the distribution—the most prototypical, high-contrast versions of a prompt—rather than the diverse, nuanced details of real images. This mode-seeking behavior is what causes the characteristic over-saturation and "cartoonish" appearance described by Alexander William Bergman and Fan Ma.

Reimagining the Distillation Objective

To fix these artifacts, researchers are moving away from the "noisy gradient" approach of standard SDS.

One powerful intuition is to treat the optimization as a path-finding problem. David McAllister and Jia-Bin Huang in Rethinking Score Distillation as a Bridge Between Image Distributions reframe SDS as a Schrödinger Bridge. By viewing the process as a transition between a "messy" source distribution and a "clean" target, they show that many artifacts come from a simple distribution mismatch that can be calibrated by better describing the artifacts themselves in the prompt.

Others have looked at the sampling math itself. Artem Lukoianov and Timur Bagautdinov argue in Score Distillation via Reparametrized DDIM that SDS is essentially a high-variance version of the DDIM sampling algorithm. By replacing random noise with noise inferred via inversion, their Score Distillation via Inversion (SDI) produces significantly sharper textures without the "deep-fried" look. Similarly, Fan Ma introduces Invariant Score Distillation (ISD) in VividDreamer, which replaces the single-step reconstruction of SDS with a multi-step refinement, effectively "denoising" the 3D model more gracefully.

Structural Anchors and Semantic Consistency

Even with perfect textures, 3D models often suffer from geometric incoherence. The 2D model doesn't "know" that a chair shouldn't have five legs when viewed from the back; it just knows what a chair looks like from some angle.

To solve the Janus problem, researchers are introducing "pseudo-3D" guidance. Kehan Li 0002 and Yanbo Fan propose P3G, which first learns a coarse, globally consistent texture to anchor the high-resolution distillation. For human avatars, Huichao Zhang and the AvatarVerse team use DensePose—a 2D-to-3D surface mapping—to force the diffusion model to respect the underlying human anatomy, preventing the generation of multiple faces.

From Pixels to Parameters

A final frontier in improving distillation is changing what we optimize. Optimizing raw pixels or voxels is inherently noisy. Kim Youwang and Tae-Hyun Oh demonstrate in Paint-it that re-parameterizing texture maps through a deep convolutional network acts as a natural filter. The network's inductive bias prevents the high-frequency noise and "burnt" colors typical of SDS, resulting in smooth, professional-grade physically-based rendering (PBR) textures.

By combining these mathematical refinements (SDI, ISD) with structural priors (AvatarVerse, HumanNorm), the field is moving from generating "hallucinated shells" toward creating high-fidelity, relightable, and geometrically sound 3D assets.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_fidelity_3d_generation_via_improved_65c796c3,
  title = {High Fidelity 3D Generation via Improved Score Distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-fidelity-3d-generation-via-improved-score-distillation/rdr_65c796c3523a482e9c53eddeb7591d6c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_fidelity_3d_generation_via_improved_65c796c3_v1,
  title = {High Fidelity 3D Generation via Improved Score Distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-fidelity-3d-generation-via-improved-score-distillation/rdr_65c796c3523a482e9c53eddeb7591d6c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}