Abstract
Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the 'incompleteness' and the 'inaccuracy' problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting.
Introduction
Multi-view Stereopsis (MVS) is the process of reconstructing a 3D model from a collection of 2D images. While most state-of-the-art MVS algorithms assume "dense" sampling—where cameras are close together with small baseline angles—real-world capture is often sparse due to cost and logistical constraints. In these "sparse" MVS scenarios, traditional methods face two critical challenges: photo-consistency failure and occlusion.
Large baseline angles cause matching patches in different images to appear significantly skewed, making it difficult for algorithms to find correspondences based on pixel similarity. Furthermore, sparse settings increase the likelihood of occlusions, where objects block a camera's view of specific points. Traditional depth-fusion methods struggle to filter these invalid views, resulting in noisy reconstructions or significant gaps in the 3D model. SurfaceNet+ addresses these issues by prioritizing geometric priors over simple photo-consistency.
Methodology
SurfaceNet+ is an end-to-end volumetric learning framework that operates directly in 3D space using voxels. By unprojecting 2D images into a 3D volume, the network implicitly handles camera geometry and avoids the skewed patch problems inherent in 2D-based depth estimation.
Multi-scale (Coarse-to-Fine) Inference
The network employs a multi-scale strategy to handle geometric inaccuracies. It begins with a coarse 3D surface prediction that acts as a geometric prior for subsequent, higher-resolution stages. This approach not only improves accuracy but also enhances efficiency; by focusing heavy computations only on sub-volumes near the predicted surface, the network achieves a 10x speedup over the original SurfaceNet.
Trainable Occlusion-aware View Selection
To resolve the incompleteness caused by occlusions, SurfaceNet+ introduces a novel volume-wise view selection mechanism. For every 3D sub-volume, the network uses the geometric prior from the previous scale to cast rays and determine if a camera's view is obstructed by the predicted surface. Invalid views are discarded, ensuring that only relevant camera pairs contribute to the final reconstruction.
Volumetric Refinement
The architecture utilizes a UNet-like 3D convolutional network to predict the probability of a voxel being on the surface. This is followed by a refinement network designed to sharpen the geometry, particularly around regions with repeating patterns.
Experiments and Results
The authors evaluated SurfaceNet+ on three major benchmarks: the DTU dataset, Tanks and Temples (T&T), and the ETH3D Low-res dataset. To simulate extreme wide-baseline conditions, they introduced a Sparse-MVS Leaderboard by subsampling these datasets to create extreme sparsity levels (e.g., taking only every 7th or 11th image).
In extreme sparse settings where baseline angles reach approximately 70°, traditional methods like COLMAP or R-MVSNet often return almost zero points. As shown in the comparison above, SurfaceNet+ maintains high recall and precision, producing complete models where other state-of-the-art methods fail. On the DTU dataset, SurfaceNet+ outperformed existing methods across all sparsity levels. Furthermore, the model demonstrated strong generalization, performing well on ETH3D and Tanks and Temples without retraining, indicating that the learned geometric features are universal.
Discussion and Future Work
While SurfaceNet+ represents a significant advancement in sparse MVS, challenges remain. Complex shading and lighting still make color-based 3D reconstruction difficult in geometrically simple regions. Additionally, while efficient, the volumetric approach faces memory hurdles when scaling to city-level reconstructions compared to point-cloud methods. Current training also requires ground-truth 3D data, such as laser scans. Future work may explore self-supervised learning to alleviate these supervision requirements.
The benchmark and implementation are publicly available at https://github.com/mjiUST/SurfaceNet-plus.
Summary updated