SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-View Stereopsis

Mengqi Ji, Jinzhi Zhang, Qionghai Dai, Lu Fang

TL;DR

SurfaceNet+ is a volumetric 3D neural network designed for Multi-view Stereopsis (MVS) in scenarios where camera views are extremely sparse. By combining a coarse-to-fine multi-scale strategy with a novel, trainable occlusion-aware view selection mechanism, it achieves high-fidelity 3D reconstruction even when traditional depth-fusion methods fail due to large baseline angles.

What Problem Does This Paper Address?

Multi-view Stereopsis (MVS) is the process of reconstructing a 3D model from a collection of 2D images. Most state-of-the-art MVS algorithms assume "dense" sampling—meaning the cameras are close together with small angles between them.

However, in the real world, capturing dense data is expensive and often impractical for large-scale scenes. When you move to "sparse" MVS (where cameras are far apart), two major problems emerge:

  1. Photo-consistency failure: Large baseline angles cause matching patches in different images to look significantly different (skewed), making it hard for algorithms to find correspondences.
  2. Occlusion: In sparse settings, it is much more likely that an object will block a camera's view of a specific point. Traditional methods struggle to filter out these "invalid" views, leading to noisy reconstructions or large holes in the model.

Methods

The authors introduce SurfaceNet+, an end-to-end volumetric learning framework. Unlike depth-map fusion methods that estimate depth for each 2D image and then merge them, SurfaceNet+ works directly in 3D space using voxels.

Key Innovations:

  • Multi-scale (Coarse-to-Fine) Inference: The network doesn't try to solve the whole scene at high resolution at once. It starts with a very coarse 3D surface prediction. This coarse model acts as a "geometric prior" for the next, more detailed stage.
  • Trainable Occlusion-aware View Selection: This is the "secret sauce." For every small 3D sub-volume (voxel group), the network decides which camera pairs are actually useful. It uses the geometric prior from the previous scale to cast "rays" and see if a camera's view is blocked by the already-predicted surface. If it is, that view is discarded for that specific sub-volume.
  • Volumetric Refinement: The network uses a UNet-like 3D convolutional architecture to predict the probability of a voxel being on the surface, followed by a refinement network to sharpen the geometry.

Datasets

The authors evaluated their model on three major benchmarks:

  1. DTU Dataset: Used for both training and primary evaluation. It contains 80 scenes with 49 camera positions.
  2. Tanks and Temples (T&T): Used to test the model on large-scale, real-world outdoor scenes.
  3. ETH3D Low-res Dataset: Used to test the model’s ability to generalize to entirely new data without fine-tuning.

Crucially, they introduced a Sparse-MVS Leaderboard by subsampling these datasets (e.g., taking only every 7th or 11th image) to simulate extreme wide-baseline conditions.

Key Insights and Ideas

  • Geometry over Photo-consistency: In sparse MVS, you can't rely solely on how similar two image patches look (photo-consistency) because the perspective shift is too great. SurfaceNet+ shows that using a rough 3D shape to guide view selection is far more robust.
  • The Power of 3D Convolutions: By unprojecting 2D images into a 3D volume, the network implicitly handles camera geometry. This avoids the "skewed patch" problem that plagues 2D-based depth estimation.
  • Efficiency through Sparsity: By using the coarse-to-fine approach, the network only needs to perform heavy computations on sub-volumes near the predicted surface. This results in a 10x speedup over the original SurfaceNet.

Key Results

  • Superior Recall: In extreme sparse settings (Sparsity = 7, where baseline angles reach ~70°), traditional methods like COLMAP or R-MVSNet often return almost zero points. SurfaceNet+ maintains high recall and precision, producing complete models where others see "black holes."
  • Benchmark Performance: On the DTU dataset, SurfaceNet+ outperformed state-of-the-art methods across all sparsity levels. Even in "dense" settings, it remains competitive, but its lead grows exponentially as the data gets sparser.
  • Generalization: The model trained on DTU performed well on the ETH3D and Tanks and Temples datasets without any retraining, proving the learned geometric features are universal.

Limitations and Future Work

  • Shading and Lighting: The authors note that for very simple geometric regions, a single view pair should suffice, but complex shading and lighting still make color-based 3D reconstruction challenging.
  • Scaling to Cities: While efficient, the volumetric approach still faces memory hurdles when scaling to massive, city-level reconstructions compared to point-cloud methods.
  • Supervision Requirements: The model currently requires ground-truth 3D data (like laser scans) for training, which can be difficult to obtain for some applications. Future work could explore self-supervised learning for sparse MVS.

Summary updated

Abstract

Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the ‘incompleteness’ and the ‘inaccuracy’ problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting.
BibTeX
@article{ji2021surfacenet,
  author = {Mengqi Ji and Jinzhi Zhang and Qionghai Dai and Lu Fang},
  title = {{SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-View Stereopsis}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=RyAOO-mzIvI},
}

Loading related content...

Figure