The ultimate goal of 3D generative modeling is to transform a simple prompt—like "a weathered leather armchair"—into a production-ready digital asset. However, a persistent "representation gap" has historically forced researchers to choose between structural integrity and visual realism. While we can generate complex skeletons or photorealistic skins, doing both simultaneously remains one of the most significant challenges in the field.
The Geometry-Appearance Gap
Most early 3D generative frameworks, such as Graph-to-3D by Fabian Manhardt et al., prioritize spatial layout and geometric shape. These models excel at understanding where a chair should sit in a room or how its legs should support its weight, but they often produce "naked" geometry—meshes or point clouds devoid of color, texture, or material properties.
This decoupling occurs because geometry and appearance are mathematically distinct. Geometry is often represented as occupancy (is a point full or empty?) or distance fields (how far am I from the surface?), while appearance is a function of light, view angle, and surface reflectance. As noted in CLIP-Forge by Aditya Sanghi et al., focusing exclusively on occupancy allows for rapid shape generation but leaves the asset unusable for creative workflows that require photorealism.
From Shape-First to Unified Diffusion
To bridge this gap, researchers are moving away from multi-step pipelines toward unified diffusion processes. The intuition is that the model should "know" what a material looks like while it is still deciding where the surface of the object lies.
SDFusion, developed by Hsin-Ying Lee and colleagues, demonstrates this by operating in a compressed latent space of Signed Distance Functions (SDFs). By leveraging 2D diffusion models to "paint" onto these 3D latents, the framework attempts to align structural generation with visual appearance. Similarly, J. Ryan Shue and Gordon Wetzstein introduced Neural Field Diffusion, which represents 3D objects as "triplanes." This representation allows the model to use powerful 2D image-generation techniques to synthesize 3D features, effectively treating the object’s appearance as an inherent part of its spatial encoding.
Refining the Surface
Even when geometry and color are generated together, achieving high-fidelity detail—like the fine grain of wood or the weave of a fabric—requires specialized refinement. PaintHuman by Jianhui Yu and Hao Zhu addresses this through Denoised Score Distillation (DSD). By using negative image-text pairs to correct optimization gradients, they prevent the "over-smoothing" that often plagues joint optimization, resulting in sharp, semantically accurate textures.
For more complex topologies, such as loose clothing or open surfaces, Surf-D by Zhengming Yu et al. utilizes Unsigned Distance Fields (UDF). This allows the model to represent non-watertight geometries while attempting to align surface geometry with consistent reflectance maps during the diffusion process.
The Challenge of Global Coherence
A recurring hurdle in this unified approach is the "Janus problem"—the tendency for models to generate multiple faces or inconsistent textures when viewed from different angles. MT3D attempts to solve this by using Deep Geometric Moments (DGM) as anchors, ensuring that as the model optimizes for appearance, it remains constrained by a consistent 3D geometric reference.
As the field matures, the focus is shifting from simply "making a shape" to "creating an asset." As Xuelin Chen argues in the CraftsMan framework, the future of 3D generation lies in mimicking the human craftsman: starting with a coarse, unified intuition of shape and color, and iteratively refining both until the geometry and its surface are indistinguishable from reality.