TL;DR
This paper provides a rigorous "reality check" for the field of 3D surface reconstruction, comparing decades of traditional optimization-based methods against the latest deep learning models. While neural networks excel when the test data matches their training, traditional methods like Screened Poisson Surface Reconstruction (SPSR) remain the champions of robustness, handling the messy noise and outliers of real-world scans far more reliably.
What Problem Does This Paper Address?
Transforming a "point cloud" (a collection of 3D coordinates) into a continuous, usable surface mesh is a fundamental but "ill-posed" problem. As shown in Figure 1, there are infinitely many surfaces that could fit a set of points.
Real-world scans are rarely perfect; they suffer from noise, outliers, non-uniform density, and missing data due to occlusions. Historically, researchers used handcrafted priors (like assuming the surface should be smooth). Recently, learning-based methods have emerged, promising to learn these priors directly from data. However, these new models are often tested on "clean" synthetic datasets, leaving a gap in understanding how they truly perform against battle-tested traditional algorithms in unpredictable, real-world scenarios.
Methods
The authors categorize surface reconstruction algorithms into a clear taxonomy based on their mathematical representation (see Figure 3):
- Surface-based methods: These directly connect points into triangles or deform a template mesh to fit the points.
- Volume-based methods: These partition 3D space into "inside" and "outside" regions, defining the surface as the boundary between them.
Within these, they distinguish between interpolation (the surface must pass exactly through the points) and approximation (the surface follows the general trend of the points). Figure 4 illustrates these different approaches visually.
The study evaluates a wide range of models, including traditional staples like SPSR and RESR, and modern neural implicit function (NIF) models like ConvONet, SAP, DGNN, and POCO.
Datasets
The authors utilize a mix of synthetic and real-world data to push these models to their limits:
- Training: Large-scale synthetic CAD databases like ShapeNet and ModelNet.
- Evaluation: The classic Berger et al. benchmark, which includes synthetic scans with varying levels of noise and outliers.
- Real-World Scans: Data from Tanks and Temples (LiDAR), DTU, and Middlebury (Multi-View Stereo).
- Novel Contribution: They implemented a synthetic scanning procedure to generate "MVS-like" point clouds, mimicking the specific occlusion and density patterns found in real image-based 3D reconstruction.
Key Insights and Ideas
- Learned Priors vs. Handcrafted Priors: Learning-based methods effectively learn "point cloud attributes" (like how to handle specific noise patterns) rather than just memorizing shapes. This allows them to outperform traditional methods by ~5% in Volumetric IoU when the test data looks like the training data.
- The Generalization Gap: The Achilles' heel of deep learning here is "out-of-distribution" data. If a model trained on simple chairs is asked to reconstruct a complex scan with high noise or outliers it hasn't seen before, its performance plummets.
- Local vs. Global: Methods that focus on local geometry (like DGNN or SAP) tend to generalize better to new shape categories than those relying on global shape embeddings.
- Traditional Robustness: Traditional methods like SPSR and RESR are remarkably resilient. Because they don't rely on training data, they don't "overfit" to specific noise levels, making them more reliable for "in-the-wild" applications.
Key Results
- In-Distribution: Learning-based methods win. POCO and SAP achieved the highest Volumetric IoU (~89%) on ShapeNet.
- Outliers: Traditional methods dominate. On datasets with 10% outliers, SPSR maintained a Volumetric IoU of ~80%, while some neural models dropped to near zero.
- Speed: SAP was the fastest overall, reconstructing a 3,000-point object in just 0.09 seconds.
- Topology: Traditional methods like RESR produced the most "compact" meshes (fewest disconnected components), whereas some neural methods struggled with "floaters" or broken surfaces when faced with noise.
Limitations and Future Work
The primary limitation identified is the sensitivity of neural models to acquisition defects. Current training sets (like ShapeNet) are too "clean" and don't represent the messy reality of 3D scanning.
The authors suggest that future research should focus on:
- Creating more diverse training datasets with realistic, sensor-specific defects.
- Developing hybrid methods that combine the robustness of traditional geometric constraints with the high precision of learned features.
- Improving the resilience of neural models to "unseen" types of noise and outliers.
The code and benchmark are available at https://github.com/raphaelsulzer/dsr-benchmark.
Summary updated