Abstract
In this work, we propose a novel approach to prioritize the depth map computation of multi-view stereo (MVS) to obtain compact 3D point clouds of high quality and completeness at low computational cost. Our prioritization approach operates before the MVS algorithm is executed and consists of two steps: finding a good set of matching partners for each view, and ranking the resulting view clusters according to their impact on quality parameters such as completeness, ground resolution, and accuracy. We utilize a novel machine learning technique for training an unsupervised confidence predictor to estimate the chances of successful depth reconstruction. In our experiments, we reach more than 70% of the maximal reachable quality fulfillment using only 5% of the available images as key views, achieving significant speedups and data reduction across different domains.
Introduction
Standard Multi-View Stereo (MVS) pipelines typically generate a depth map for every single image in a dataset. While this ensures completeness, it creates a massive computational and storage bottleneck. For datasets with hundreds of high-resolution images, this can result in billions of 3D points that must be stored, filtered, and fused.
A fundamental redundancy exists in modern photo-capture: many images observe the same parts of a scene with similar geometry. However, simple geometric heuristics (like triangulation angles) often fail because MVS is an "ill-posed" task affected by textureless surfaces, moving objects, or complex occlusions. This research aims to identify a minimal subset of "view clusters" that provide the highest quality reconstruction at the lowest possible cost.
Methodology
The proposed prioritization approach operates before the MVS algorithm is executed, utilizing a two-step pipeline:
- Matching Partner Selection: For every "key view" (an image used to generate a depth map), the system identifies the best $k$ matching partners. This selection uses a combination of geometric constraints and a learned MVS confidence measure.
- Next-Best View (NBV) Ranking: View clusters are ranked based on their predicted impact on a "quality fulfillment function," accounting for scene coverage, ground resolution, 3D uncertainty, and reconstruction likelihood.
The Confidence Predictor
The core innovation is a machine learning model that predicts the probability of successful depth reconstruction for every pixel. This model is trained unsupervised, using the results of depth map fusion from previous runs as a supervision signal. If a point survives fusion, it is labeled a "success." The ranking problem is formulated as a monotone submodular function, allowing for a greedy optimization strategy with mathematical guarantees of near-optimality.
Experimental Setup
The approach was evaluated on two distinct scenarios:
- Valley Dataset: 1,236 images of prehistoric rock art in Italy, featuring high-accuracy laser scanner ground truth.
- House Dataset: 485 images of a suburban house, used to test domain generalization for a model trained on natural rock surfaces.
Results and Discussion
The prioritization scheme demonstrates extreme efficiency in identifying high-value viewpoints. On the Valley dataset, the method achieved 70% quality fulfillment using only 5% of available images as key views.
Key performance metrics include:
- Data Reduction: A 20x reduction in the number of generated 3D points (430M vs. 8.6B).
- Computational Speedup: A 10x speedup in depth map computation.
- Generalization: The model successfully generalized to the "House" dataset, outperforming standard geometric baselines even in an unseen domain.
- Efficiency: The ranking process itself is lightweight, taking only 2-6% of the time required for MVS matching due to pre-computed visibility and confidence caching.
Conclusion
This work highlights the significant redundancy in modern MVS datasets and provides a robust framework for prioritizing computation. While the method depends on an initial sparse reconstruction (SfM), its ability to achieve high-quality results with a fraction of the data offers a path toward more efficient 3D reconstruction pipelines, potentially enabling real-time feedback during image acquisition.
Summary updated