Efficient Edge-Preserving Multi-View Stereo Network for Depth Estimation

Wanjuan Su, Wenbing TaoAAAI 2023 · DOI

TL;DR

This paper introduces EPNet, a novel multi-view stereo (MVS) network that tackles the common problem of blurry depth maps at object boundaries. By incorporating a hierarchical edge-preserving residual learning module and a cross-view photometric consistency loss, EPNet achieves state-of-the-art accuracy while maintaining high efficiency, outperforming existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Reconstructing accurate 3D scenes from multiple images is a fundamental task in computer vision, known as Multi-View Stereo (MVS). While learning-based MVS methods have made significant strides, they often struggle with a critical issue: producing blurry depth maps, especially around thin structures and object boundaries. This blurriness arises from two main culprits: the inherent smoothing properties of 3D Convolutional Neural Networks (CNNs) used for cost volume regularization, and the discrete, sparse sampling of potential depth values. Existing "coarse-to-fine" architectures, which progressively refine depth estimates, exacerbate this problem. Errors in the initial, low-resolution depth estimates are propagated and amplified in subsequent stages, and simple upsampling techniques introduce further artifacts. This not only degrades the visual quality of the depth maps but also hinders the creation of accurate 3D point clouds. Furthermore, many high-performing MVS methods are computationally expensive, limiting their practical applicability.

Methods

EPNet introduces two key innovations to address these challenges:

  • Hierarchical Edge-Preserving Residual Learning (HEPR) Module: Instead of relying solely on image content, HEPR leverages context information from the reference image to refine depth estimates. It works by learning "residual maps" that capture high-frequency details. This module is integrated hierarchically within the network, progressively rectifying upsampling errors and refining multi-scale depth estimations. This approach allows for edge-aware upsampling, ensuring that the refined depth maps retain sharp details and align better with image edges.
  • Cross-view Photometric Consistency (CPC) Loss: To further enhance the accuracy of detailed structures, the CPC loss is introduced. This loss function measures the difference between synthesized images generated from the ground truth depth and the estimated depth, but in the image domain rather than the depth domain. This clever shift magnifies minor errors in detailed regions of the depth map, making them more prominent for optimization, while suppressing large errors from texture-less areas. This effectively guides the network to focus on improving fine-grained details.

These modules are embedded within a lightweight cascade framework. This framework employs a coarse-to-fine strategy but with a focus on efficiency. It uses a fixed resolution across stages, allowing for a higher number of depth hypotheses to be sampled without a significant increase in memory or computational cost.

Datasets

The paper evaluates EPNet on several standard MVS benchmarks:

  • DTU dataset: A widely used dataset for MVS research, featuring diverse scenes and lighting conditions.
  • BlendedMVS dataset: A large-scale dataset with a variety of scenes.
  • Tanks and Temples dataset: Includes both realistic outdoor and indoor scenes, with intermediate and advanced subsets.
  • ETH3D high-res benchmark: A challenging dataset with high-resolution images and wider baseline views.

The authors also mention using the DTU dataset for training and fine-tuning on BlendedMVS.

Key Insights and Ideas

  • Image context is crucial for edge preservation: The HEPR module demonstrates that incorporating image context beyond just geometric consistency is vital for producing sharp, edge-aligned depth maps.
  • Image-domain loss for better detail refinement: Shifting the photometric consistency loss to the image domain (CPC loss) is a powerful technique to amplify and correct errors in fine-grained regions that might be overlooked in a depth-domain loss.
  • Fixed-resolution cascade for efficiency: By keeping the resolution consistent across cascade stages, the authors achieve a better trade-off between the number of depth hypotheses (accuracy) and computational resources (efficiency).
  • Hierarchical refinement is more effective: Applying edge-preserving refinement at intermediate stages, rather than just as a final post-processing step, leads to more robust and accurate depth estimation throughout the network.

Key Results

EPNet achieves state-of-the-art performance on multiple benchmarks:

  • Accuracy: EPNet demonstrates competitive accuracy, often outperforming existing methods, particularly on challenging datasets like Tanks and Temples and ETH3D. For instance, on the Tanks and Temples advanced subset, it achieves the best result among published works.
  • Efficiency: The proposed lightweight cascade framework and HEPR module lead to significant reductions in GPU memory consumption (up to 88.90% reduction compared to some methods) and run-time (up to 74.00% reduction). EPNet's memory and run-time are comparable to other efficient methods, but with superior generalization.
  • Qualitative Results: Visual comparisons show that EPNet recovers finer details and sharper object boundaries compared to methods like CasMVSNet and UniMVSNet. The depth maps produced by EPNet align much better with the input images.

Limitations and Future Work

  • Limitations: While the paper highlights significant improvements, the authors acknowledge that "blur artifacts not only reduce the accuracy of the depth map but also reduce the completeness of the reconstructed point cloud by destroying the geometric consistency between views." This suggests that while EPNet mitigates blur, perfect reconstruction of all geometric details might still be an ongoing challenge. The paper also notes that "some efficient methods are proposed recently... they show unpleasant performance compared with non-efficient methods." This implies a general trade-off between efficiency and performance that EPNet aims to balance, but it's a persistent challenge in the field.
  • Future Work: The paper doesn't explicitly detail future work directions. However, the success of EPNet in achieving both high accuracy and efficiency suggests that further exploration of hierarchical edge-preserving techniques and efficient network architectures for MVS is a promising avenue. The focus on improving detail recovery and generalization also points towards continued research in these areas.

Summary updated

BibTeX
@inproceedings{su2023efficient,
  author = {Wanjuan Su and Wenbing Tao},
  title = {{Efficient Edge-Preserving Multi-View Stereo Network for Depth Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25330},
}
Figure