The Decoupling of Shape and Texture in 3D Generation

72 concepts67 papers1 proposal

In the pursuit of creating photorealistic 3D assets, research has historically split the problem into two distinct domains: geometry (the structural scaffold) and appearance (the surface texture, color, and material properties). While generative models have become remarkably adept at "sculpting" complex 3D shapes, they often leave these objects as "gray clay," requiring external tools or secondary processes to apply visual details.

This decoupling is not merely a matter of convenience; it reflects a fundamental challenge in how we represent and learn 3D data.

The Geometric Scaffold

Most modern 3D generative frameworks focus exclusively on spatial occupancy or distance fields. For example, Sanghi et al. (CLIP-Forge) and Zheng (LAS-Diffusion) demonstrate how to generate high-fidelity shapes from text or sketches by predicting Signed Distance Fields (SDFs) or point clouds. Similarly, Yu et al. (Surf-D) use Unsigned Distance Fields to represent complex, non-watertight geometries like clothing.

These methods excel at capturing structural topology, but as Zhang et al. (3DILG) and Zheng et al. (ImFace) observe, these representations are often "appearance-blind." They lack the native capacity to store or synthesize the intricate reflectance maps or albedo needed for photorealism. Even state-of-the-art mesh generators like Tan et al. (CraftsMan3D) treat high-fidelity texture generation as a separate, open challenge, focusing their diffusion processes entirely on geometric refinement.

Why Texture is Hard to Integrate

The difficulty in joint geometry-texture synthesis stems from three primary hurdles:

  1. Disentanglement: As Henderson et al. argue, learning from 2D images requires the model to separate the object's true color from the lighting conditions of the scene. Without this disentanglement, the model might "bake" shadows or highlights directly into the texture.
  2. Representation Mismatch: Geometry is often represented by continuous functions (like SDFs) or sparse structures (like point clouds), while textures are traditionally 2D maps. Mapping a 2D texture onto a complex 3D surface—a process known as UV mapping—is notoriously difficult to automate. Liu et al. (TexOct) attempt to bypass this by using octree-based diffusion to synthesize textures directly on the surface, but this still requires a pre-existing mesh.
  3. Global vs. Local Consistency: Kim (SyncTweedies) highlights that while generating a single consistent image is easy, maintaining a seamless 360-degree texture that respects the underlying 3D structure is incredibly difficult for standard diffusion models.

Bridging the Gap with Geometric Cues

To move beyond "gray clay," researchers are exploring ways to ground appearance in geometric reality. One intuitive approach is to use surface normals—mathematical descriptions of surface orientation—to guide the generative process.

Jain et al. (OrienText) demonstrate this by using normals to ensure that generated text follows the physical contours of an object, preventing the "flat sticker" look common in earlier models like Geng et al. (AnyText2). Similarly, Oxholm et al. showed early on that geometric normal fields are far more stable for texture analysis than fickle 2D appearance data.

Toward Unified Asset Creation

The ultimate goal is a pipeline where shape and material co-evolve. The authors of SOPHY argue that modeling shape and material independently overlooks their inherent interdependence—for instance, a heavy stone texture is physically inconsistent with a thin, wire-like geometry.

Recent efforts like Cui et al. (NeuSDFusion) suggest that integrating spatially-aware texture branches directly into the 3D diffusion process may be the key to ensuring that the "look" of an object is as structurally sound as its "shape." Until these domains are fully unified, the generation of production-ready 3D assets will remain a two-step process of sculpting followed by painting.

Key Foundational Work

Large Scale 3D Morphable Models (2018) — contributed 1 concept elements to this direction.
The Scale of Geometric Texture (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_surface_appearance_modeling_b63011dd,
  title = {surface appearance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surface-appearance-modeling/rdr_b63011dd3b9f4b20ba644af20bc50629},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_surface_appearance_modeling_b63011dd_v1,
  title = {surface appearance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surface-appearance-modeling/rdr_b63011dd3b9f4b20ba644af20bc50629/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake