Just a Few Points are All You Need for Multi-view Stereo: A Novel Semi-supervised Learning Method for Multi-view Stereo

Taekyung Kim, Jaehoon Choi, Seokeon Choi, Dongki Jung, Changick Kim

Abstract

Multi-view Stereo (MVS) typically demands dense 3D ground truth for training, which is costly and difficult to acquire. This paper presents SGT-MVSNet, a semi-supervised framework capable of high-quality 3D reconstruction using as few as 30-40 ground truth points per scene—a $10^5$ reduction in labeling requirements compared to traditional supervised methods. By leveraging a dual-prediction architecture, 3D point consistency loss, and a coarse-to-fine depth propagation module, SGT-MVSNet achieves performance comparable to fully supervised methods and demonstrates strong generalization to unseen datasets.

Introduction

Multi-view Stereo (MVS) is the process of reconstructing dense 3D geometry from a collection of 2D images. While deep learning has significantly advanced MVS, state-of-the-art models are notoriously "data-hungry," requiring dense depth maps or 3D point clouds for supervision. Collecting such ground truth (GT) is a resource-intensive process involving expensive LiDAR sensors, extensive scanning time, and manual post-processing to handle dynamic objects. Consequently, models trained on limited datasets like DTU often struggle to generalize to real-world environments. SGT-MVSNet addresses this bottleneck by asking: Can we train a high-quality MVS model using only a tiny fraction of 3D points?

Methodology

The authors propose SGT-MVSNet (Sparse Ground truth-based MVS Network), which bridges the gap between sparse labels and dense reconstruction through three primary innovations:

  1. Double Prediction Layers: The network architecture incorporates a main prediction layer and an auxiliary half-resolution layer. This dual structure enables the model to encode richer contextual information, which is critical when explicit depth labels are sparse.
  2. 3D Point Consistency Loss (PCL): This self-supervision mechanism ensures that depth predictions are consistent across different views. A predicted depth point in one image is back-projected into 3D space and then re-projected into another view. To ensure reliability, a probability map acts as a confidence score, restricting the loss to non-occluded, high-confidence regions.
  3. Coarse-to-fine Reliable Depth Propagation (CRDP): To handle challenging areas like object edges and occlusions where consistency loss may fail, the CRDP module propagates high-confidence depth values from "easy" regions to "hard" regions based on local feature similarity and dilated convolutions.
The SGT-MVSNet architecture, featuring dual prediction layers and the propagation module.

Experimental Results and Insights

The model was evaluated on the DTU Dataset using a "Sparse GT" configuration, where ground truth points were randomly sampled at a ratio of $1 \times 10^{-5}$ (approximately 30-40 points per scene).

Performance Benchmarks

SGT-MVSNet significantly outperformed standard supervised models when trained on the same sparse data. On the DTU dataset, it achieved an Overall score of 0.411mm, compared to 0.612mm for Point-MVSNet. Remarkably, the performance of SGT-MVSNet trained on sparse data is comparable to models trained on full, dense ground truth.

Generalization and Logic

The model demonstrated robust generalization on the Tanks and Temples dataset without additional fine-tuning. A key insight from the study is that MVS networks possess an inherent "systematic depth reasoning" capability. Sparse supervision acts as a set of "anchor points," allowing the network to propagate depth information across the scene using feature matching and geometric consistency.

3D reconstruction results demonstrating high completeness and detail on complex scenes.

Conclusion and Future Work

SGT-MVSNet proves that dense supervision is not a prerequisite for high-quality MVS. While the model excels in completeness, there remains a slight trade-off in point-wise accuracy compared to some specialized supervised models. Additionally, the propagation module can occasionally introduce blurring at depth boundaries. Future research may explore combining this sparse-GT approach with fully unsupervised photometric consistency cues to further refine edge precision and eliminate the need for any ground truth labels.

Summary updated

Abstract

While learning-based multi-view stereo (MVS) methods have recently shown successful performances in quality and efficiency, limited MVS data hampers generalization to unseen environments. A simple solution is to generate various large-scale MVS datasets, but generating dense ground truth for 3D structure requires a huge amount of time and resources. On the other hand, if the reliance on dense ground truth is relaxed, MVS systems will generalize more smoothly to new environments. To this end, we first introduce a novel semi-supervised multi-view stereo framework called a Sparse Ground truth-based MVS Network (SGT-MVSNet) that can reliably reconstruct the 3D structures even with a few ground truth 3D points. Our strategy is to divide the accurate and erroneous regions and individually conquer them based on our observation that a probability map can separate these regions. We propose a self-supervision loss called the 3D Point Consistency Loss to enhance the 3D reconstruction performance, which forces the 3D points back-projected from the corresponding pixels by the predicted depth values to meet at the same 3D co-ordinates. Finally, we propagate these improved depth pre-dictions toward edges and occlusions by the Coarse-to-fine Reliable Depth Propagation module. We generate the spare ground truth of the DTU dataset for evaluation and extensive experiments verify that our SGT-MVSNet outperforms the state-of-the-art MVS methods on the sparse ground truth setting. Moreover, our method shows comparable reconstruction results to the supervised MVS methods though we only used tens and hundreds of ground truth 3D points.
BibTeX
@inproceedings{kim2021just,
  author = {Taekyung Kim and Jaehoon Choi and Seokeon Choi and Dongki Jung and Changick Kim},
  title = {{Just a Few Points are All You Need for Multi-view Stereo: A Novel Semi-supervised Learning Method for Multi-view Stereo}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00612},
}
Figure