Abstract
In this paper, we introduce Point2Mesh, a technique for reconstructing a surface mesh from an input point cloud. Instead of explicitly specifying a prior that encodes the expected shape properties, the prior is defined automatically using the input point cloud, which we refer to as a self-prior. The self-prior encapsulates reoccurring geometric repetitions from a single shape within the weights of a deep neural network. We optimize the network weights to deform an initial mesh to shrink-wrap a single input point cloud. This explicitly considers the entire reconstructed shape, since shared local kernels are calculated to fit the overall object. The convolutional kernels are optimized globally across the entire shape, which inherently encourages local-scale geometric self-similarity across the shape surface. We show that shrink-wrapping a point cloud with a self-prior converges to a desirable solution; compared to a prescribed smoothness prior, which often becomes trapped in undesirable local minima. While the performance of traditional reconstruction approaches degrades in non-ideal conditions that are often present in real world scanning, i.e., unoriented normals, noise and missing (low density) parts, Point2Mesh is robust to non-ideal conditions. We demonstrate the performance of Point2Mesh on a large variety of shapes with varying complexity.
Introduction
Reconstructing a continuous surface mesh from a discrete, noisy point cloud is a fundamental but ill-posed problem in computer graphics. Real-world 3D scans are rarely perfect; they often contain noise, missing regions (holes), and lack oriented normals. Traditional methods, such as Poisson Surface Reconstruction, rely on manually designed priors like smoothness, which frequently fail when data is sparse or normals are inconsistent. Conversely, supervised deep learning approaches require massive datasets of ground-truth 3D models. Point2Mesh bridges this gap by utilizing a "self-prior"—a deep neural network optimized on a single shape at inference time—making it robust to non-ideal scanning conditions without requiring external training data.
Methodology
The core innovation of Point2Mesh is the Self-Prior. Inspired by the "Deep Image Prior" in 2D imaging, the architecture of a Convolutional Neural Network (CNN) acts as a regularizer.
The Optimization Loop
Instead of training a network to generalize across many objects, Point2Mesh optimizes the weights of a CNN to deform an initial mesh (typically a convex hull) to fit a specific input point cloud. The system utilizes MeshCNN, which applies convolutions directly to the edges of a mesh. Because convolutional kernels are shared across the entire shape, the network is forced to find self-similar geometric features, effectively "learning" the recurring patterns of the specific object.
Coarse-to-Fine Strategy and Loss Functions
Reconstruction occurs in stages, starting with a low-resolution mesh and a small number of points. The mesh is iteratively upsampled to capture finer details. The optimization is guided by a bi-directional Chamfer Distance and a novel Beam-Gap Loss, which helps the surface reach into deep, narrow cavities. To handle high-resolution meshes, the system supports part-based optimization.
Evaluation Datasets
The authors evaluated Point2Mesh on several standard benchmarks, including Thingi10k, COSEG, TOSCA, and the Surface Reconstruction Benchmark (Berger et al. 2013), as well as real-world scans captured with a NextEngine 3D laser scanner.
Key Insights and Results
The primary insight is that CNNs are naturally adept at representing patterns but poor at representing noise. Shared weights (kernels) prefer to explain a shape using repeating local structures. Since noise is random, the CNN-based prior naturally filters it out.
Point2Mesh consistently outperforms traditional and other learning-based methods in denoising and hole-filling tasks. It guarantees a watertight, manifold surface by deforming a single continuous mesh. Furthermore, it is agnostic to normals, working effectively with raw, unoriented point samples—a significant advantage over traditional algorithms.
Limitations and Conclusion
While powerful, Point2Mesh is computationally intensive, as it requires a full optimization for every new shape. Runtimes can range from minutes to several hours. Additionally, the method depends on an initial mesh; while a convex hull suffices for many objects, complex topologies (high genus) require more sophisticated initialization. Future work may involve automatic genus detection to improve starting mesh selection and exploring the framework for finding correspondences between different 3D objects.
Summary updated