A Survey and Benchmark of Automatic Surface Reconstruction from Point Clouds

Raphael Sulzer, Renaud Marlet, Bruno Vallet, Loic Landrieu

TL;DR

This paper provides a rigorous "reality check" for the field of 3D surface reconstruction, comparing decades of traditional optimization-based methods against the latest deep learning models. While neural networks excel when the test data matches their training, traditional methods like Screened Poisson Surface Reconstruction (SPSR) remain the champions of robustness, handling the messy noise and outliers of real-world scans far more reliably.

What Problem Does This Paper Address?

Transforming a "point cloud" (a collection of 3D coordinates) into a continuous, usable surface mesh is a fundamental but "ill-posed" problem. As shown in Figure 1, there are infinitely many surfaces that could fit a set of points.

Challenges in surface reconstruction: unknown topology, geometry, and acquisition defects.

Real-world scans are rarely perfect; they suffer from noise, outliers, non-uniform density, and missing data due to occlusions. Historically, researchers used handcrafted priors (like assuming the surface should be smooth). Recently, learning-based methods have emerged, promising to learn these priors directly from data. However, these new models are often tested on "clean" synthetic datasets, leaving a gap in understanding how they truly perform against battle-tested traditional algorithms in unpredictable, real-world scenarios.

Methods

The authors categorize surface reconstruction algorithms into a clear taxonomy based on their mathematical representation (see Figure 3):

  • Surface-based methods: These directly connect points into triangles or deform a template mesh to fit the points.
  • Volume-based methods: These partition 3D space into "inside" and "outside" regions, defining the surface as the boundary between them.
Figure 3: Survey overview: Surface reconstruction algorithms can be categorized based on their primary representation approach: surface-based or volume-based. Within each category, methods are further classified as either interpolation- based or approximation-based. Subsequently, specific subgroups and various individual methods are identified under each classification, providing a comprehensive framework for understanding the landscape of surface reconstruction techniques.

Within these, they distinguish between interpolation (the surface must pass exactly through the points) and approximation (the surface follows the general trend of the points). Figure 4 illustrates these different approaches visually.

Figure 4: Examples of surface reconstruction algorithms: In (a), we exemplify a reconstruction from a surface-based interpolation approach. Close by points are connected with triangles to reconstruct the surface. In (b), we show a reconstruction of a surface-based approximation approach. An initial mesh, e.g., a plane, is deformed to fit to the input points. In (c), we show a volume-based interpolation. Input points are connected using a 3D Delaunay tetrahedralization. The resulting tetrahedra are then labelled as inside (dark blue) or outside the surface (red) and the interface triangles constitute the final mesh. In (d), we show a volume-based approximation. The surface is represented as an appropriate level set of an occupancy function.

The study evaluates a wide range of models, including traditional staples like SPSR and RESR, and modern neural implicit function (NIF) models like ConvONet, SAP, DGNN, and POCO.

Datasets

The authors utilize a mix of synthetic and real-world data to push these models to their limits:

  • Training: Large-scale synthetic CAD databases like ShapeNet and ModelNet.
  • Evaluation: The classic Berger et al. benchmark, which includes synthetic scans with varying levels of noise and outliers.
  • Real-World Scans: Data from Tanks and Temples (LiDAR), DTU, and Middlebury (Multi-View Stereo).
  • Novel Contribution: They implemented a synthetic scanning procedure to generate "MVS-like" point clouds, mimicking the specific occlusion and density patterns found in real image-based 3D reconstruction.

Key Insights and Ideas

  • Learned Priors vs. Handcrafted Priors: Learning-based methods effectively learn "point cloud attributes" (like how to handle specific noise patterns) rather than just memorizing shapes. This allows them to outperform traditional methods by ~5% in Volumetric IoU when the test data looks like the training data.
  • The Generalization Gap: The Achilles' heel of deep learning here is "out-of-distribution" data. If a model trained on simple chairs is asked to reconstruct a complex scan with high noise or outliers it hasn't seen before, its performance plummets.
  • Local vs. Global: Methods that focus on local geometry (like DGNN or SAP) tend to generalize better to new shape categories than those relying on global shape embeddings.
  • Traditional Robustness: Traditional methods like SPSR and RESR are remarkably resilient. Because they don't rely on training data, they don't "overfit" to specific noise levels, making them more reliable for "in-the-wild" applications.

Key Results

  • In-Distribution: Learning-based methods win. POCO and SAP achieved the highest Volumetric IoU (~89%) on ShapeNet.
  • Outliers: Traditional methods dominate. On datasets with 10% outliers, SPSR maintained a Volumetric IoU of ~80%, while some neural models dropped to near zero.
  • Speed: SAP was the fastest overall, reconstructing a 3,000-point object in just 0.09 seconds.
  • Topology: Traditional methods like RESR produced the most "compact" meshes (fewest disconnected components), whereas some neural methods struggled with "floaters" or broken surfaces when faced with noise.

Limitations and Future Work

The primary limitation identified is the sensitivity of neural models to acquisition defects. Current training sets (like ShapeNet) are too "clean" and don't represent the messy reality of 3D scanning.

The authors suggest that future research should focus on:

  1. Creating more diverse training datasets with realistic, sensor-specific defects.
  2. Developing hybrid methods that combine the robustness of traditional geometric constraints with the high precision of learned features.
  3. Improving the resilience of neural models to "unseen" types of noise and outliers.

The code and benchmark are available at https://github.com/raphaelsulzer/dsr-benchmark.

Summary updated

Abstract

We present a comprehensive survey and benchmark of both traditional and learning-based methods for surface reconstruction from point clouds. This task is particularly challenging for real-world acquisitions due to factors such as noise, outliers, non-uniform sampling, and missing data. Traditional approaches often simplify the problem by imposing handcrafted priors on either the input point clouds or the resulting surface, a process that can require tedious hyperparameter tuning. In contrast, deep learning models have the capability to directly learn the properties of input point clouds and desired surfaces from data. We study the influence of handcrafted and learned priors on the precision and robustness of surface reconstruction techniques. We evaluate various time-tested and contemporary methods in a standardized manner. When both trained and evaluated on point clouds with identical characteristics, the learning-based models consistently produce higher-quality surfaces compared to their traditional counterparts -- even in scenarios involving novel shape categories. However, traditional methods demonstrate greater resilience to the diverse anomalies commonly found in real-world 3D acquisitions. For the benefit of the research community, we make our code and datasets available, inviting further enhancements to learning-based surface reconstruction. This can be accessed at https://github.com/raphaelsulzer/dsr-benchmark .
BibTeX
@article{sulzer2023survey,
  author = {Raphael Sulzer and Renaud Marlet and Bruno Vallet and Loic Landrieu},
  title = {{A Survey and Benchmark of Automatic Surface Reconstruction from Point Clouds}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
}
Figure