Abstract
Multi-view Stereo (MVS) typically demands dense 3D ground truth for training, which is costly and difficult to acquire. This paper presents SGT-MVSNet, a semi-supervised framework capable of high-quality 3D reconstruction using as few as 30-40 ground truth points per scene—a $10^5$ reduction in labeling requirements compared to traditional supervised methods. By leveraging a dual-prediction architecture, 3D point consistency loss, and a coarse-to-fine depth propagation module, SGT-MVSNet achieves performance comparable to fully supervised methods and demonstrates strong generalization to unseen datasets.
Introduction
Multi-view Stereo (MVS) is the process of reconstructing dense 3D geometry from a collection of 2D images. While deep learning has significantly advanced MVS, state-of-the-art models are notoriously "data-hungry," requiring dense depth maps or 3D point clouds for supervision. Collecting such ground truth (GT) is a resource-intensive process involving expensive LiDAR sensors, extensive scanning time, and manual post-processing to handle dynamic objects. Consequently, models trained on limited datasets like DTU often struggle to generalize to real-world environments. SGT-MVSNet addresses this bottleneck by asking: Can we train a high-quality MVS model using only a tiny fraction of 3D points?
Methodology
The authors propose SGT-MVSNet (Sparse Ground truth-based MVS Network), which bridges the gap between sparse labels and dense reconstruction through three primary innovations:
- Double Prediction Layers: The network architecture incorporates a main prediction layer and an auxiliary half-resolution layer. This dual structure enables the model to encode richer contextual information, which is critical when explicit depth labels are sparse.
- 3D Point Consistency Loss (PCL): This self-supervision mechanism ensures that depth predictions are consistent across different views. A predicted depth point in one image is back-projected into 3D space and then re-projected into another view. To ensure reliability, a probability map acts as a confidence score, restricting the loss to non-occluded, high-confidence regions.
- Coarse-to-fine Reliable Depth Propagation (CRDP): To handle challenging areas like object edges and occlusions where consistency loss may fail, the CRDP module propagates high-confidence depth values from "easy" regions to "hard" regions based on local feature similarity and dilated convolutions.
Experimental Results and Insights
The model was evaluated on the DTU Dataset using a "Sparse GT" configuration, where ground truth points were randomly sampled at a ratio of $1 \times 10^{-5}$ (approximately 30-40 points per scene).
Performance Benchmarks
SGT-MVSNet significantly outperformed standard supervised models when trained on the same sparse data. On the DTU dataset, it achieved an Overall score of 0.411mm, compared to 0.612mm for Point-MVSNet. Remarkably, the performance of SGT-MVSNet trained on sparse data is comparable to models trained on full, dense ground truth.
Generalization and Logic
The model demonstrated robust generalization on the Tanks and Temples dataset without additional fine-tuning. A key insight from the study is that MVS networks possess an inherent "systematic depth reasoning" capability. Sparse supervision acts as a set of "anchor points," allowing the network to propagate depth information across the scene using feature matching and geometric consistency.
Conclusion and Future Work
SGT-MVSNet proves that dense supervision is not a prerequisite for high-quality MVS. While the model excels in completeness, there remains a slight trade-off in point-wise accuracy compared to some specialized supervised models. Additionally, the propagation module can occasionally introduce blurring at depth boundaries. Future research may explore combining this sparse-GT approach with fully unsupervised photometric consistency cues to further refine edge precision and eliminate the need for any ground truth labels.
Summary updated