In 3D computer vision, researchers have long faced a trade-off between implicit representations (like Signed Distance Fields) and explicit representations (like polygonal meshes). Implicit fields are mathematically elegant and handle topology changes easily, but they are "black boxes" that are difficult to render efficiently or edit in standard graphics software. Explicit meshes are the industry standard for animation and games, but their discrete nature—specifically the fixed connectivity of vertices—makes them notoriously difficult to optimize using gradient-based deep learning.
A growing body of research has bridged this gap by developing differentiable surface extraction layers. These techniques allow a neural network to output a continuous field and then "reach through" the extraction process to optimize the resulting mesh directly.
Making the Discrete Differentiable
The classic algorithm for turning a volumetric grid into a mesh is Marching Cubes. However, Marching Cubes is inherently non-differentiable because it involves discrete look-up tables and thresholding. To solve this, Yiyi Liao and Andreas Geiger introduced Deep Marching Cubes, which predicts occupancy probabilities and vertex displacements. By making the mapping from voxels to mesh vertices differentiable, the network can optimize geometric losses directly on the final surface rather than on intermediate voxel grids.
An alternative approach, proposed by Edoardo Remelli in MeshSDF, decouples the forward and backward passes. While the forward pass uses standard, non-differentiable algorithms to extract the mesh, the backward pass injects custom gradients derived from functional analysis. This allows the system to backpropagate errors from 2D silhouettes or 3D points directly to the underlying Signed Distance Function (SDF), enabling the model to "pull" the surface toward the correct shape.
Hybrid Representations and Tetrahedral Grids
While voxel grids are simple, they scale poorly with resolution. Recent work has shifted toward tetrahedral grids to represent geometry more efficiently. In MagicPony, Shangzhe Wu and colleagues use a differentiable "marching tetrahedra" layer. This hybrid approach defines geometry as an SDF on a tetrahedral grid, which is then converted into an explicit mesh. This allows the model to handle topological changes—like a horse's legs moving apart—while maintaining the rendering speed of a mesh.
Similarly, Xiaoke Huang developed EMA for human avatars, which uses a differentiable conversion from implicit fields to tetrahedral meshes. This strategy allows the model to optimize continuous geometric representations while remaining fully compatible with standard mesh-based graphics pipelines for real-time rendering.
End-to-End Rendering and Refinement
The ultimate goal of these differentiable layers is to enable analysis-by-synthesis: optimizing a 3D shape so that its 2D rendering matches an input image. MeshLRM by Kai Zhang and Hao Su integrates differentiable mesh extraction directly into a transformer-based "Large Reconstruction Model." By supervising the final mesh with a ray opacity loss, the model learns to produce high-quality geometry and textures in a single feed-forward pass.
For high-fidelity human reconstruction, Peng Li and Yan-Pei Cao introduced PSHuman, which uses explicit remeshing. Instead of relying purely on implicit fields, they deform a base template using differentiable rasterization. This allows the model to "carve" fine details, such as clothing wrinkles, directly into the mesh surface by optimizing against predicted normal maps.
Overcoming Topological Constraints
One of the hardest problems in mesh optimization is changing the "connectivity"—the graph of how vertices are linked. Sanghyun Son and Yang Zhou addressed this in DMesh by using power diagrams. This allows for nearly linear-time differentiable triangulation, enabling gradient-based optimization to change a shape's topology without the computational overhead of traditional volumetric methods.
Other researchers have explored unconventional slicing methods. Chenbo Jiang and colleagues proposed NeuralSlice, which represents 3D objects as the intersection of a 4D tetrahedral mesh and a 4D hyperplane. This differentiable slicing operation bypasses Marching Cubes entirely, producing manifold meshes 10x faster than state-of-the-art implicit methods.
Unsigned Fields and Open Surfaces
Most differentiable meshing focuses on "watertight" volumes, but real-world objects like clothing or leaves are often thin, open surfaces. Benoît Guillard and Pascal Fua developed MeshUDF to handle Unsigned Distance Fields (UDFs). By extending Marching Cubes to reason about gradients rather than sign flips, they enabled the differentiable reconstruction of complex, non-watertight geometries that were previously inaccessible to end-to-end neural pipelines.