The field of 3D computer graphics is experiencing a profound paradigm split. On one side, modern deep learning models excel at generating and reconstructing 3D scenes using implicit representations (such as Neural Radiance Fields and Signed Distance Functions) or unstructured primitives (such as point clouds and 3D Gaussian splats). On the other side, the entire commercial graphics ecosystem—including game engines, physics simulators, and animation pipelines—is built exclusively around explicit triangle meshes.
This disconnect creates a significant representation-pipeline incompatibility. While neural representations are highly continuous and differentiable, making them ideal for gradient descent, they lack the explicit topology, connectivity, and clean surface definitions required by industry-standard software. Bridging this gap is one of the most critical challenges in modern 3D vision.
The Anatomy of the Connectivity Gap
To understand why this gap exists, we must look at how neural models represent space versus how traditional graphics engines render it.
Unstructured Primitives (Points and Splats)
Point clouds and 3D Gaussian splats represent 3D shapes as a collection of disconnected elements floating in space. As A. Manessis observed in early work on scene reconstruction, sparse point data completely lacks topological information; there is no inherent rule dictating which points should be connected to form a solid surface.
Kostiuk et al. note in HyperColor that converting these point sets into functional assets requires computationally heavy post-processing. This issue has intensified with the advent of 3D Gaussian Splatting. As reviewed by Xie et al. in Advancing Extended Reality with 3D Gaussian Splatting: Innovations and Prospects, extracting clean, manifold surfaces from millions of overlapping ellipsoidal primitives remains an open challenge because the splats do not naturally map to continuous geometric boundaries.
Continuous Implicit Fields
Representations like Neural Radiance Fields (NeRFs) avoid discrete points by encoding a scene as a continuous volumetric density field. However, turning a fuzzy cloud of probabilities into a sharp polygon edge is notoriously difficult.
Tsalicoglou et al. explain in TextMesh that standard extraction techniques, such as Marching Cubes, frequently produce noisy, non-manifold, or fragmented surfaces when applied to density fields. Because the underlying model does not explicitly optimize for a hard boundary, Goel et al. demonstrate in Differentiable Stereopsis that volumetric representations often fail to enforce the geometric coherence required for practical asset creation, resulting in floating artifacts and hollow geometries.
[Neural Optimization] [Traditional Graphics]
Continuous Fields / Splats ──(The Gap)──> Explicit Triangle Meshes
(Easy to differentiate) (Easy to rasterize & animate)
Why Naive Conversion Fails
A common misconception is that the connectivity gap can be solved simply by running a surface reconstruction algorithm (like Poisson Reconstruction or Marching Cubes) as a final post-processing step. In practice, this naive decoupling fails for two primary reasons:
- Loss of Appearance and Materials: Traditional surface extraction algorithms operate purely on geometry, ignoring how light interacts with the surface. Jon Hasselgren demonstrates in Appearance-Driven Automatic 3D Model Simplification that converting implicit representations without accounting for appearance results in a severe degradation of visual fidelity. To preserve details, the geometry and material textures (such as normal maps) must be optimized jointly.
- Unusable Topology: Standard extraction tools generate millions of tiny, disorganized triangles. These over-tessellated meshes are impossible for technical artists to clean, unwrap for UV texturing, or rig for skeletal animation. As emphasized by Xu et al. in DreamFace and Sanyal et al. in SCULPT, production environments require clean, structured topologies that can deform naturally during physics simulations or character animations.
Strategies for Bridging the Gap
To resolve this incompatibility, researchers are moving away from passive post-processing and are instead designing architectures that bake explicit mesh constraints directly into the neural learning process.
Direct Mesh Generation
One prominent strategy is to bypass intermediate volumetric representations entirely. By training models to output mesh structures directly, the network learns to mimic the clean topology of human-authored assets.
Siddiqui et al. pioneer this approach with MeshGPT, which treats triangle meshes as sequences of geometric "words." By training a decoder-only transformer on a vocabulary of mesh faces, the model generates compact, clean topologies directly. Similarly, Jun Gao et al. introduce GET3D, which embeds a differentiable surface modeling layer (DMTET) directly into a generative adversarial network, allowing it to output industry-standard textured meshes directly from 2D image collections.
Joint Geometry and Appearance Optimization
Another paradigm uses differentiable rendering to optimize an explicit mesh template so that its rendered appearance matches the target neural representation or source images.
Yang et al. implement this in DreamMesh by jointly manipulating the vertices and texturing of a triangle mesh during text-to-3D generation, completely avoiding the "fuzzy" geometry typical of volumetric methods. Chen et al. employ a similar philosophy in Fantasia3D by strictly decoupling geometry learning from physically-based rendering (PBR) material modeling. Because the geometry is represented explicitly during training, the resulting asset can be instantly exported into standard graphics engines for realistic relighting and interaction.
Hybrid Representations
Rather than forcing a choice between the photorealism of neural fields and the utility of meshes, some frameworks maintain a dual representation.
For dynamic and changing environments, Li et al. introduce DreamMesh4D, which links an explicit triangular mesh with a Gaussian Splatting field via a hybrid skinning algorithm. This allows the system to leverage the rapid, high-fidelity rendering of splats while remaining fully compatible with the bone-based animation pipelines standard in game development.