Geometry-Aware Diffusion for 3D Texture Synthesis

55 concepts44 papers1 proposal

Generating realistic textures for 3D objects involves a fundamental tension: 2D generative models, such as Stable Diffusion, possess immense "knowledge" of visual appearance but lack an inherent understanding of 3D structure. When these models are applied naively to 3D meshes, they often produce "texture drifting," where patterns slide across the surface, or the "Janus problem," where the model hallucinates multiple faces or front-facing features on the back of an object.

To bridge this gap, a growing body of research focuses on geometry-aware diffusion. The core intuition is to treat the 3D geometry not just as a canvas, but as a structural anchor that constrains the generative process. By conditioning 2D diffusion models on geometric maps—such as depth, surface normals, or dense coordinate mappings—researchers ensure that synthesized textures are spatially consistent, semantically aligned, and physically plausible.

Geometry as a Structural Anchor

The most direct way to ground a generative model in 3D space is through depth-conditioned guidance. In this paradigm, a 2D diffusion model is not asked to create an image from scratch; instead, it is given a depth map rendered from a specific viewpoint of the 3D mesh.

As Haibo Yang et al. demonstrate in 3DStyle-Diffusion, integrating depth maps via architectures like ControlNet allows for fine-grained control over materials. The depth map acts as a spatial contract: the diffusion process is forced to adhere to the object's profile, preventing the hallucination of features in anatomically incorrect locations. Similarly, Dave Zhenyu Chen et al. show in Text2Tex that using depth as a structural prior effectively eliminates the "stretching" artifacts that occur when 2D textures are projected onto complex 3D curvatures.

This structural anchoring is particularly vital for human avatars. Huichao Zhang argues in AvatarVerse that moving from sparse skeletal signals to dense surface mappings (like DensePose) provides the model with a precise coordinate-aware map. This ensures that features like buttons or pockets are placed correctly relative to the body's geometry, effectively resolving the multi-face artifacts common in earlier text-to-3D methods.

Semantic Alignment via Surface Normals

While depth maps provide a silhouette and basic volume, they often lack the high-frequency detail needed for complex textures. To capture how light interacts with a surface or how a pattern should follow a curve, researchers incorporate surface normals and curvature.

In TextureDreamer, Jia-Bin Huang et al. integrate normal maps into the diffusion process to ensure that generated patterns follow the physical curves of the object. This prevents a "sticker-like" appearance where textures look flat and disconnected from the underlying shape. Yawar Siddiqui et al. further demonstrate in Texturify that intrinsic properties like curvature are essential for semantic alignment—allowing a model to "know" that the slats of a chair or the lights of a car should align with specific geometric edges.

This principle extends to facial reconstruction. Alexandros Lattas et al. show in AvatarMe++ that using high-frequency geometric information guides the inference of skin pores and fine wrinkles, ensuring these details are spatially consistent with the underlying 3D facial structure.

The Generate-then-Refine Strategy

Because 3D texturing often requires consistency across multiple viewpoints, many frameworks adopt a multi-stage optimization process. The geometry serves as the consistent thread between these stages.

  1. Initial Synthesis: A base texture is generated using global geometric priors.
  2. Refinement: The model performs localized "image-to-image" tasks on rendered views, conditioned on depth to maintain alignment.

Christina Tsalicoglou et al. utilize this in TextMesh, where a depth-conditioned refinement stage bypasses the over-saturation common in standard score distillation sampling. By treating texture generation as a conditioned task, the model can focus on high-fidelity details without losing the global structure. Jimin Xu et al. take this further in MPOD123 by using 2D masks as geometric anchors, ensuring that fine-grained textures are injected precisely within a consistent outline.

Applications in Reconstruction and Editing

The utility of geometry-aware diffusion extends beyond creation to the restoration and editing of 3D data. When dealing with incomplete 3D scans, Julian Chibane et al. demonstrate that using the reconstructed geometry as a conditioning signal provides a structural prior for color distribution, preventing color bleeding across distinct geometric boundaries.

In the realm of editing, Mengting Chen et al. show that integrating depth maps into the generative process allows for "imitative editing." This decoupling of shape and appearance ensures that when a user transfers a texture from a reference image to a target object, the new visual patterns conform perfectly to the target's original 3D structure, preserving its physical integrity.

By treating geometry as a first-class citizen in the diffusion process, these methods move the field toward a more robust synthesis of the 3D world, where appearance and structure are inextricably linked.

Key Foundational Work

Context-aware textures (2007) — contributed 2 concept elements to this direction.
Symmetry-Aware Mesh Segmentation into Uniform Overlapping Patches (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_conditioned_texture_synthesis_1aaaaa3a,
  title = {depth-conditioned texture synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-conditioned-texture-synthesis/rdr_1aaaaa3af2d24215bf8075bdb60ad497},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_conditioned_texture_synthesis_1aaaaa3a_v1,
  title = {depth-conditioned texture synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-conditioned-texture-synthesis/rdr_1aaaaa3af2d24215bf8075bdb60ad497/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake