MeshUDF: Fast and Differentiable Meshing of Unsigned Distance Field Networks

Benoit Guillard, Federico Stella, Pascal Fua

TL;DR

MeshUDF introduces a fast, accurate, and differentiable extension of the classic Marching Cubes algorithm designed specifically for Unsigned Distance Fields (UDFs). This allows researchers to extract high-quality explicit 3D meshes from neural networks representing complex, non-watertight shapes (like open clothing or car interiors) in seconds rather than hours. Crucially, by making the meshing process differentiable, it enables direct gradient-based optimization to fit 3D shapes to sparse data like point clouds or 2D silhouettes.

What Problem Does This Paper Address?

Deep implicit representations, such as Signed Distance Fields (SDFs) and occupancy fields, have revolutionized 3D shape modeling. However, they suffer from a fundamental constraint: they can only represent watertight geometries (shapes with a clearly defined "inside" and "outside"). This leaves out a massive class of real-world objects, such as open garments (shirts, skirts), zero-thickness shells, or objects with complex inner structures (like cars with seats).

Unsigned Distance Fields (UDFs) can represent these non-watertight surfaces by predicting the absolute distance to the closest surface point. The catch? Converting a UDF back into an explicit 3D triangle mesh has traditionally been a major bottleneck. Existing pipelines either "inflate" the surface to force it to be watertight (which artificially thickens the geometry and ruins accuracy) or extract a dense point cloud and use slow, sequential algorithms like Ball-Pivoting to stitch it together. Furthermore, none of these UDF meshing techniques are differentiable, meaning you cannot backpropagate errors from the final mesh back into the neural network to fit sparse observations.

Methods

MeshUDF bridges this gap by modifying the traditional Marching Cubes algorithm to work with unsigned fields and analytical gradients.

  1. Gradient-Based Pseudo-Signs: Standard Marching Cubes relies on sign flips (positive to negative) across grid cell corners to detect where a surface cuts through an edge. Since UDF values are strictly positive, MeshUDF looks at the gradients instead. If the gradients at two neighboring grid corners point in opposite directions, it indicates a surface lies between them. The authors use this intuition to define a "pseudo-signed distance."
  2. Breadth-First Voting Scheme: Neural UDFs can be noisy. To prevent inconsistent facet orientations and random holes, the algorithm traverses the 3D grid using a breadth-first search. It employs a voting scheme where neighboring grid points collaborate to robustly determine the pseudo-sign of a cell corner.
  3. Differentiable Iso-Surface Extraction: To make the vertex positions differentiable with respect to the network's latent shape vector ($z$), the authors look at $\alpha$-levelsets slightly away from the surface.
  4. Boundary Handling: For open boundaries (edges of the garment), standard normal-based derivatives fail because they don't allow the surface to shrink or expand laterally. MeshUDF introduces a novel boundary derivative formulation using a perpendicular vector ($\bo$) mapped along a semicircle, allowing the mesh boundaries to naturally expand or contract during gradient descent.
   Standard SDF Meshing: Uses Sign Flips (+ / -) -> Watertight Only
   MeshUDF (Our Method): Uses Gradient Directions (→ / ←) -> Non-Watertight & Differentiable

Datasets

The authors evaluated their approach using two primary datasets:

  • Multi-Garment Net (MGN): A dataset containing 328 digital clothing meshes (shirts, pants, skirts). The authors used 300 for training an auto-decoder network and 28 for testing.
  • ShapeNet (Cars): A subset of 300 car models used to test how well MeshUDF generalizes when plugged downstream of a publicly available, pretrained Neural Distance Field (NDF) network.

Key Insights and Ideas

  • Gradients Substitute for Signs: The core "aha moment" is that gradient orientation contains the exact topological information that UDFs lack due to the absence of a sign.
  • Unlocking the Boundaries: By treating open boundaries differently than the interior of the surface (using the semicircle mapping direction $\bo$), the authors solved a long-standing issue in differentiable rendering: allowing open-surface meshes to change their surface area dynamically via pure gradient descent.
  • End-to-End Compatibility: Because the meshing operation outputs analytical derivatives, it can be seamlessly paired with differentiable mesh renderers (like PyTorch3D) to optimize 3D shapes directly from 2D image silhouettes.

Key Results

  • Massive Speedups: For garment meshing, MeshUDF takes just 1.2 seconds, compared to the Ball-Pivoting baseline which takes over 50 minutes (16.5s for point sampling + 3000s for triangulation). For ShapeNet cars, it reduces the time from over 2 hours down to 7.1 seconds.
  • Higher Accuracy: MeshUDF achieves a lower Chamfer Distance (CHD) and better Normal Consistency (NC) than the Inflation baseline because it extracts the true zero-thickness surface rather than an artificially thickened shell.
  • Effective Sparse Fitting: When optimizing a random latent code to match a sparse point cloud (only 200 points) or a single 2D silhouette, optimizing via the MeshUDF explicit mesh yielded significantly better accuracy than trying to optimize purely within the implicit UDF domain.

Limitations and Future Work

  • Dependency on UDF Quality: The algorithm assumes the underlying neural network predicts a clean, sharp 0-levelset. If the network has low capacity, the UDF can "thicken" into a volume, causing staircase artifacts or single-cell holes.
  • Marching Cubes Constraints: The method inherits standard Marching Cubes limitations, including cubic memory scalability relative to grid resolution and an inability to natively handle complex self-intersections or non-orientable surfaces (like Möbius strips) without producing holes.
  • Future Directions: The authors suggest developing network initialization or regularization strategies to train UDF networks completely from scratch using these explicit mesh gradients, as well as applying the framework to jointly track human pose and loose clothing from video.

Summary updated

Abstract

Unsigned Distance Fields (UDFs) can be used to represent non-watertight surfaces. However, current approaches to converting them into explicit meshes tend to either be expensive or to degrade the accuracy. Here, we extend the marching cube algorithm to handle UDFs, both fast and accurately. Moreover, our approach to surface extraction is differentiable, which is key to using pretrained UDF networks to fit sparse data.
BibTeX
@inproceedings{guillard2022meshudf,
  author = {Benoit Guillard and Federico Stella and Pascal Fua},
  title = {{MeshUDF: Fast and Differentiable Meshing of Unsigned Distance Field Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20062-5\_33},
}
Figure