Prioritized Multi-View Stereo Depth Map Generation Using Confidence Prediction

Christian Mostegel, Friedrich Fraundorfer, Horst BischofarXiv 2018 · OpenReview

Abstract

In this work, we propose a novel approach to prioritize the depth map computation of multi-view stereo (MVS) to obtain compact 3D point clouds of high quality and completeness at low computational cost. Our prioritization approach operates before the MVS algorithm is executed and consists of two steps: finding a good set of matching partners for each view, and ranking the resulting view clusters according to their impact on quality parameters such as completeness, ground resolution, and accuracy. We utilize a novel machine learning technique for training an unsupervised confidence predictor to estimate the chances of successful depth reconstruction. In our experiments, we reach more than 70% of the maximal reachable quality fulfillment using only 5% of the available images as key views, achieving significant speedups and data reduction across different domains.

Introduction

Standard Multi-View Stereo (MVS) pipelines typically generate a depth map for every single image in a dataset. While this ensures completeness, it creates a massive computational and storage bottleneck. For datasets with hundreds of high-resolution images, this can result in billions of 3D points that must be stored, filtered, and fused.

A fundamental redundancy exists in modern photo-capture: many images observe the same parts of a scene with similar geometry. However, simple geometric heuristics (like triangulation angles) often fail because MVS is an "ill-posed" task affected by textureless surfaces, moving objects, or complex occlusions. This research aims to identify a minimal subset of "view clusters" that provide the highest quality reconstruction at the lowest possible cost.

Methodology

The proposed prioritization approach operates before the MVS algorithm is executed, utilizing a two-step pipeline:

  1. Matching Partner Selection: For every "key view" (an image used to generate a depth map), the system identifies the best $k$ matching partners. This selection uses a combination of geometric constraints and a learned MVS confidence measure.
  2. Next-Best View (NBV) Ranking: View clusters are ranked based on their predicted impact on a "quality fulfillment function," accounting for scene coverage, ground resolution, 3D uncertainty, and reconstruction likelihood.
Overview of the proposed view cluster prioritization pipeline, from input data to ranked view clusters.

The Confidence Predictor

The core innovation is a machine learning model that predicts the probability of successful depth reconstruction for every pixel. This model is trained unsupervised, using the results of depth map fusion from previous runs as a supervision signal. If a point survives fusion, it is labeled a "success." The ranking problem is formulated as a monotone submodular function, allowing for a greedy optimization strategy with mathematical guarantees of near-optimality.

Experimental Setup

The approach was evaluated on two distinct scenarios:

  • Valley Dataset: 1,236 images of prehistoric rock art in Italy, featuring high-accuracy laser scanner ground truth.
  • House Dataset: 485 images of a suburban house, used to test domain generalization for a model trained on natural rock surfaces.

Results and Discussion

The prioritization scheme demonstrates extreme efficiency in identifying high-value viewpoints. On the Valley dataset, the method achieved 70% quality fulfillment using only 5% of available images as key views.

Visual comparison of reconstruction quality fulfillment at 30%, 50%, 70%, and 80% levels, showing the progressive completion of the scene with increasing key views.

Key performance metrics include:

  • Data Reduction: A 20x reduction in the number of generated 3D points (430M vs. 8.6B).
  • Computational Speedup: A 10x speedup in depth map computation.
  • Generalization: The model successfully generalized to the "House" dataset, outperforming standard geometric baselines even in an unseen domain.
  • Efficiency: The ranking process itself is lightweight, taking only 2-6% of the time required for MVS matching due to pre-computed visibility and confidence caching.

Conclusion

This work highlights the significant redundancy in modern MVS datasets and provides a robust framework for prioritizing computation. While the method depends on an initial sparse reconstruction (SfM), its ability to achieve high-quality results with a fraction of the data offers a path toward more efficient 3D reconstruction pipelines, potentially enabling real-time feedback during image acquisition.

Summary updated

Abstract

In this work, we propose a novel approach to prioritize the depth map computation of multi-view stereo (MVS) to obtain compact 3D point clouds of high quality and completeness at low computational cost. Our prioritization approach operates before the MVS algorithm is executed and consists of two steps. In the first step, we aim to find a good set of matching partners for each view. In the second step, we rank the resulting view clusters (i.e. key views with matching partners) according to their impact on the fulfillment of desired quality parameters such as completeness, ground resolution and accuracy. Additional to geometric analysis, we use a novel machine learning technique for training a confidence predictor. The purpose of this confidence predictor is to estimate the chances of a successful depth reconstruction for each pixel in each image for one specific MVS algorithm based on the RGB images and the image constellation. The underlying machine learning technique does not require any ground truth or manually labeled data for training, but instead adapts ideas from depth map fusion for providing a supervision signal. The trained confidence predictor allows us to evaluate the quality of image constellations and their potential impact to the resulting 3D reconstruction and thus builds a solid foundation for our prioritization approach. In our experiments, we are thus able to reach more than 70% of the maximal reachable quality fulfillment using only 5% of the available images as key views. For evaluating our approach within and across different domains, we use two completely different scenarios, i.e. cultural heritage preservation and reconstruction of single family houses.
BibTeX
@misc{mostegel2018prioritized,
  author = {Christian Mostegel and Friedrich Fraundorfer and Horst Bischof},
  title = {{Prioritized Multi-View Stereo Depth Map Generation Using Confidence Prediction}},
  year = {2018},
  eprint = {1803.08323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.08323},
}
Figure