NeuralSlice: Neural 3D Triangle Mesh Reconstruction via Slicing 4D Tetrahedral Meshes

Chenbo Jiang, Jie Yang, Shwai He, Yu-Kun Lai, Lin GaoICML 2023 · OpenReview

Introduction

Learning-based, high-fidelity reconstruction of 3D shapes with varying topology is a fundamental challenge in computer vision and graphics. Historically, researchers have had to choose between two distinct shape representations: explicit meshes or implicit functions. NeuralSlice introduces a novel paradigm that bridges this gap. By representing 3D objects as the intersection of a deformable 4D tetrahedral mesh and a 4D hyperplane, NeuralSlice achieves the topological flexibility of implicit functions alongside the computational efficiency of explicit meshes. The result is direct manifold mesh generation that operates up to 10x faster than state-of-the-art implicit methods.

The Problem: Explicit vs. Implicit Representations

In 3D deep learning, the choice of shape representation dictates the capabilities and limitations of a model:

  • Explicit Representations (Meshes): These methods output vertices and faces directly. They are computationally efficient, easy to render, and preserve thin structures well. However, because they typically deform a fixed template (like a sphere), their connectivity is rigid. Reconstructing a shape with a different topology—such as morphing a sphere into a torus—requires messy "tearing" operations.
  • Implicit Representations (SDFs/UDFs): These methods represent shapes as level sets of a continuous function. They handle complex topologies and arbitrary holes beautifully. Unfortunately, they are computationally heavy. Extracting a usable 3D mesh requires sampling thousands of points in a dense grid and running the Marching Cubes algorithm, creating a massive performance bottleneck.

NeuralSlice aims to deliver the best of both worlds: an explicit method that outputs a mesh directly, yet remains topologically flexible enough to represent holes, handles, and disjointed parts.

The NeuralSlice Approach

The core inspiration for NeuralSlice comes from level set theory and the geometry of conic sections. Just as slicing a 3D cone with a 2D plane can produce a circle, an ellipse, or a hyperbola depending on the angle, slicing a 4D object with a 3D hyperplane can produce 3D meshes with entirely different topologies.

The Pipeline

  1. Feature Extraction: The model processes an input point cloud using a Global Encoder (PointNet) and a Local Encoder to construct a rich 3D feature volume.
  2. 4D Template Deformation: The system initializes a 4D tetrahedral mesh template. An Embedding Net maps these 4D coordinates into the 3D feature volume to extract local geometric features. A Deformation Net then predicts how to spatially adjust these 4D vertices based on the extracted features.
  3. Learnable Slicing: The network predicts a scalar parameter $\alpha$, which defines a 4D hyperplane ($w = \alpha$).
  4. The Slice Operation: The deformed 4D mesh is intersected with the predicted hyperplane. The authors introduce a differentiable method to calculate these intersection points, which directly form the vertices and faces of the final 3D triangle mesh.
  5. Refinement: To capture high-frequency details, a subdivision-based refinement network adds vertices and fine-tunes their 3D spatial positions.

To ensure the resulting mesh is clean and well-behaved, the model employs Lipschitz normalization to bound deformations and Laplacian regularization to maintain smooth, regular triangles.

Key Insights and Innovations

  • Topology through Uplifting: By elevating the 3D reconstruction problem into 4D space, NeuralSlice bypasses the "fixed connectivity" trap. A single 4D mesh can represent a chair, a table, or a lamp simply by altering its deformation and the angle of the slicing plane.
  • Guaranteed Manifolds: The authors mathematically prove that, under mild conditions, their slicing operation is guaranteed to produce a manifold triangle mesh. This is critical for downstream applications like physics simulation or 3D printing.
  • Efficiency via Explicitness: Because NeuralSlice calculates intersections directly rather than regressing a dense grid of occupancy values, it entirely avoids the Marching Cubes sampling bottleneck.
  • Thin Structure Preservation: Implicit methods often blur or break thin structures (like chair legs) due to grid resolution limits. As a mesh-based approach, NeuralSlice preserves these delicate structures robustly.

Performance and Results

The authors evaluated NeuralSlice on the ShapeNet dataset across 13 common categories, using standard benchmark splits for fair comparison.

  • Accuracy: NeuralSlice achieved lower Chamfer Distance (CD) scores compared to explicit baselines like NFM and AtlasNet, showing particular strength on shapes with complex topologies.
  • Speed: During the reconstruction phase, NeuralSlice is roughly 10x faster than ConvOcc (a leading implicit method) because it skips the expensive mesh extraction step.
  • Representation Power: The model's flexibility was showcased by demonstrating that a single 4D tetrahedral mesh can represent an entire continuous deformation sequence—such as a sphere smoothly morphing into a cow—simply by moving the slicing plane through the 4D volume.

Limitations and Future Directions

While NeuralSlice represents a significant leap forward, the authors acknowledge a few limitations. As an explicit method, it can occasionally struggle to recover extremely fine, high-frequency details compared to SDF-based methods, which benefit from dense distance-field supervision. Additionally, designing the initial 4D templates is mathematically and computationally non-trivial.

Looking ahead, the authors suggest extending this framework to 4D game engines. Currently, "4D shapes" in gaming are built by hand and remain relatively simple. NeuralSlice could pave the way for automating the generation of complex, dynamic 4D assets, opening new doors for interactive media and procedural generation.

Summary updated

Abstract

Learning-based high-fidelity reconstruction of 3D shapes with varying topology is a fundamental problem in computer vision and computer graphics. Recent advances in learning 3D shapes using explici...
BibTeX
@inproceedings{jiang2023neuralslice,
  author = {Chenbo Jiang and Jie Yang and Shwai He and Yu-Kun Lai and Lin Gao},
  title = {{NeuralSlice: Neural 3D Triangle Mesh Reconstruction via Slicing 4D Tetrahedral Meshes}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=HGVnUS6OYN},
}
Figure