TL;DR
This paper introduces DELS-MVS, a novel deep learning approach for Multi-View Stereo (MVS) that directly searches for corresponding pixels along epipolar lines, avoiding the need for depth range discretization and costly cost volumes. This method achieves competitive results on standard MVS benchmarks by iteratively refining depth estimates and employing a geometry-aware fusion strategy guided by learned confidence maps.
What Problem Does This Paper Address?
Multi-View Stereo (MVS) aims to reconstruct dense 3D depth maps from multiple calibrated images of the same scene. While deep learning has significantly improved MVS performance, existing methods often rely on depth-plane sweeping cost volumes. These approaches suffer from high memory footprints and computational costs, and crucially, require pre-defining a depth range and discretizing it. This discretization can be suboptimal, leading to uneven sampling of the epipolar line and potentially missing accurate matches, especially in scenes with large depth variations or unknown depth ranges.
Methods
The core innovation of DELS-MVS is its Deep Epipolar Line Search (DELS). Instead of discretizing depth, DELS operates directly on the epipolar line for each pixel in the reference image.
- Iterative Epipolar Residual Search: For each pixel, the algorithm iteratively estimates an "epipolar residual" that corrects an initial guess. This is framed as a classification problem where a deep neural network (ER-Net) predicts which segment of the epipolar line contains the correct match.
- Coarse-to-Fine Strategy: The search is performed across multiple resolution levels (full, half, quarter) and within each level, the search space (partition width) is dynamically refined. This coarse-to-fine approach allows for efficient exploration of the epipolar line.
- No Depth Range Discretization: A key advantage is that DELS-MVS does not require a predefined minimum or maximum depth range, making it more robust to varying scene scales.
- Confidence Network (C-Net): A separate network estimates a confidence map for each depth map generated from a source image. This confidence is crucial for the subsequent fusion step.
- Geometry-Aware Fusion: The method fuses the individual depth maps from multiple source images using a geometry-aware strategy. It prioritizes depth candidates with high confidence and narrow depth ranges, leveraging the learned confidences to filter outliers and promote robustness.
Datasets
- Training: DTU dataset [1] is used for training the ER-Net and C-Net.
- Evaluation: The method is evaluated on three standard MVS benchmarks:
- ETH3D High-Res [22]: A high-resolution benchmark.
- Tanks and Temples [11]: Evaluated on both intermediate and advanced sets.
- DTU [1]: Standard benchmark for MVS evaluation.
- Fine-tuning: For evaluation on ETH3D and Tanks and Temples, the network is fine-tuned on the Blended MVS dataset [38].
Key Insights and Ideas
- Epipolar Line as the Search Space: Directly searching along the epipolar line, rather than discretizing depth, is a more natural and efficient way to find correspondences, especially when depth ranges are unknown or vary significantly.
- Iterative Classification for Residuals: Framing the epipolar residual estimation as an iterative classification problem (predicting the correct segment) allows for a robust, coarse-to-fine search without explicit depth discretization.
- Learned Confidence for Robust Fusion: The confidence network is vital for effectively fusing multiple depth maps. By learning which depth estimates are reliable, the system can better handle occlusions and varying scene geometry.
- Geometry-Aware Fusion: Explicitly incorporating geometric relationships and confidence scores during fusion leads to more accurate and complete reconstructions.
Key Results
- Competitive Performance: DELS-MVS achieves competitive results across all evaluated benchmarks (ETH3D, Tanks and Temples, DTU), often outperforming or matching state-of-the-art methods.
- ETH3D: Shows superior completeness and less noise compared to EPP-MVSNet [17].
- DTU: Achieves improved AVG scores compared to recent methods like IterMVS [26] and EPP-MVSNet [17], and competitive AVG results with Uni-MVSNet [20] and Trans-MVSNet [28].
- Efficiency: The method avoids building large cost volumes, leading to reduced memory and computational costs compared to traditional deep MVS methods. For example, on DTU, a 1600x1184 depth map requires 2.77s and 6.0GB of memory.
Limitations and Future Work
- Parameter Tuning: The paper mentions several parameters (e.g., confidence thresholds $\beta$, $\eta$, $\omega$, and iteration counts $I_j$) that require careful tuning.
- Source Image Selection: While the method uses N source images, the selection of these images is not explicitly discussed as a learned component, though the fusion strategy implicitly handles varying quality.
- Future Work: The authors suggest that further improvements could be made by exploring more advanced fusion strategies or by incorporating more explicit geometric reasoning into the network architecture. The paper also notes that the dynamic partition update and the use of outer partitions for occlusion modeling are key to its success, suggesting these are areas for further exploration.
Summary updated