Normal Map Guidance for 3D Diffusion Priors

17 concepts10 papers1 proposal

When we lift 2D diffusion models to 3D generation and reconstruction, we encounter a fundamental bottleneck: 2D images conflate geometry with appearance. A standard RGB image bundles shape, surface texture, and environmental lighting into a single grid of pixels.

When a 3D optimization process (such as Score Distillation Sampling) relies solely on RGB images, the underlying generative model often takes a shortcut. Instead of carving out complex 3D structures, it satisfies the 2D prompt by "painting" high-contrast textures and shadows onto flat or incorrect geometry. This shortcut is the root cause of common artifacts like baked-in lighting, over-saturation, and the multi-faced "Janus" problem.

To solve this, a powerful paradigm has emerged: Normal Map Guidance. By forcing diffusion models to operate on surface normal maps rather than RGB images, researchers decouple geometry from appearance, providing a pure structural prior that directly shapes the 3D world.


The Intuition: Why Normal Maps Outperform RGB and Depth

To understand why normal maps are uniquely suited for 3D guidance, consider what a surface normal map actually represents. It maps the local 3D orientation vector $(n_x, n_y, n_z)$ of a surface directly to the RGB channels of a 2D image.

[Flat Surface + Painted Texture] ---> RGB Diffusion: "Looks like a wrinkle!"
                                 ---> Normal Map:   [Uniform Color] (No actual geometry)

[Deformed Surface (True Wrinkle)] ---> Normal Map:   [Gradient Shift] (True geometric orientation)

If an optimization algorithm attempts to cheat by painting a clothing wrinkle onto a flat mesh, an RGB diffusion model might be fooled. However, a normal map rendering of that flat mesh remains completely uniform. The cheat is immediately exposed.

As established by Ignatyev et al., normal maps provide a "purer geometric signal" because they strip away the distractions of color and illumination. Furthermore, normal maps are superior to raw 3D coordinates or depth maps for 2D generative models. As shown by Huang et al., mapping 3D surface normals into the RGB color space provides a highly effective pixel-aligned signal. This allows standard 2D diffusion architectures to natively interpret local surface orientation and accurately predict missing or occluded geometry from novel viewpoints.


Disentangled Score Distillation

The most direct application of this intuition is in geometry optimization. In a typical text-to-3D pipeline, a 3D representation (like a mesh or a neural field) is rendered from a random viewpoint, and a 2D diffusion model provides gradients to update the 3D parameters.

In the framework proposed by Chen et al., this process is entirely decoupled. Instead of rendering RGB images, the pipeline renders normal maps and passes them to a diffusion model trained to understand surface orientations. Because the latent space of the diffusion model is forced to evaluate structural encoding alone, the optimization gradients act directly on the spatial derivatives of the 3D shape.

3D Representation (e.g., DMTET) 
       │
       ▼ (Render)
  Normal Map 
       │
       ▼ 
 ┌──────────────────────────┐
 │ Normal-Aware Diffusion   │ ◄─── Text Prompt ("A wrinkled jacket")
 └──────────────────────────┘
       │
       ▼ (Score Distillation Gradients)
 Updates 3D Geometry Directly (Forces physical deformation)

This mechanism forces the optimization to develop high-frequency surface details and smoother topology. For instance, Liu et al. integrated normal maps into score distillation sampling to produce assets with intricate geometric details that are completely free of "baked-in" shadows, making them fully relightable in downstream graphics engines. Similarly, in sparse-view human reconstruction, Chen et al. demonstrated that optimizing based on surface orientation rather than color ensures that the model recovers actual 3D structures—such as fine clothing wrinkles—that are typically lost or flattened by standard texture-based optimization.


Conditioning and Preventing Structural Drift

Beyond optimization, normal maps serve as an exceptional inductive bias for feed-forward view synthesis and surface completion.

When generating novel views of a complex object (like a human head), relying solely on camera pose embeddings can lead to "structural drift," where the geometry morphs uncharacteristically across views. Tang et al. addressed this by conditioning a multi-view diffusion model on rendered normal maps. This provides a pixel-aligned geometric guide that forces synthetic "hallucinations" to remain strictly anchored to the target 3D structure.

To make this possible, the underlying diffusion models must be adapted to speak the language of geometry:

  • Fine-Tuning: Frameworks like the one presented by Shao et al. fine-tune diffusion models directly on massive datasets of surface normals and depth maps, bridging the gap between 2D image generation and explicit 3D shape priors.
  • Personalized LoRAs: For specialized tasks like virtual try-ons, Xie et al. utilized Low-Rank Adaptations (LoRAs) trained specifically on normal maps to enhance the model's precision when generating complex garment geometry.

By synthesizing or conditioning on precise surface normals alongside RGB images—as also explored by Comas Massague et al. and Galanakis et al.—these systems achieve strict 3D consistency and realistic view invariance. The model is no longer guessing what an object looks like from the side; it is adhering to an explicit, mathematically rigorous map of surface orientations.

Key Foundational Work

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023) — contributed 2 concept elements to this direction.
UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation (2024) — contributed 2 concept elements to this direction.
NeuSD: Surface Completion With Multi-View Text-to-Image Diffusion (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_normal_map_guidance_4696ab1c,
  title = {normal map guidance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/normal-map-guidance/rdr_4696ab1c19a741c9a7ceeba7d3354a0e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_normal_map_guidance_4696ab1c_v1,
  title = {normal map guidance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/normal-map-guidance/rdr_4696ab1c19a741c9a7ceeba7d3354a0e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake