Multi-View Stereo Vision Patchmatch Algorithm Based on Data Augmentation

Feiyang Pan, Pengtao Wang, Lin Wang, Lihong LiSensors 2023 · DOI

TL;DR

This paper introduces a novel multi-view stereo (MVS) algorithm, DA-PatchmatchNet, that leverages data augmentation and an efficient patchmatch-based approach to reconstruct 3D models from images. By reducing computational memory and runtime compared to traditional 3D cost volume methods, it enables processing of higher-resolution images, making it suitable for resource-constrained platforms.

What Problem Does This Paper Address?

Reconstructing accurate 3D models from multiple 2D images (Multi-View Stereo or MVS) is a challenging task in computer vision. Traditional MVS methods often struggle with issues like image occlusion, varying lighting conditions, and non-textured areas. More recent deep learning-based MVS methods, while improving accuracy, often rely on computationally expensive 3D convolutions to build and regularize a "cost volume" that represents depth possibilities. This high computational and memory demand limits their application to high-resolution images and resource-constrained devices. The paper aims to bridge this gap by proposing an MVS algorithm that is both accurate and efficient, particularly for higher-resolution inputs.

Methods

The proposed algorithm, DA-PatchmatchNet, combines a learning-based patchmatch approach with data augmentation. Key components include:

  • Data Augmentation Branch: This branch enhances the network's robustness by artificially altering input images. Techniques include adjusting brightness, contrast, and hue, and randomly masking out parts of the image. This helps the model generalize better and become less sensitive to variations in lighting and occlusions.
  • Depth Estimation Branch: This branch handles the core 3D reconstruction.
  • Multi-Scale Feature Extraction: Similar to Feature Pyramid Networks (FPN), it extracts image features at different scales, feeding them into a coarse-to-fine processing pipeline.
  • Patchmatch Algorithm: Instead of relying solely on 3D convolutions for cost volume regularization, the algorithm incorporates a learned patchmatch strategy. This involves:
  • Initialization and Local Perturbation: An initial depth map is randomly sampled, and then iteratively refined with local perturbations.
  • Adaptive Propagation: This technique uses learned offsets to propagate depth information from neighboring pixels, capturing spatial correlations more effectively than simple fixed propagation.
  • Adaptive Evaluation: This involves feature mapping from source images to depth layers and a cost matching step that fuses features using learned pixel view weights.
  • Adaptive Spatial Cost Aggregation: This refines the cost aggregation process by considering spatial similarities and depth hypothesis similarities.
  • Dynamic Interval d: The algorithm uses a dynamic sampling interval for depth layers, adapting it based on pixel differences across scales. This helps preserve more image information compared to fixed sampling intervals.
  • Depth Regression: Finally, a softmax function converts the aggregated cost volume into a confidence volume, from which the final depth map is regressed.

The algorithm replaces the memory-intensive 3D convolutions with adaptive evaluation and aggregation, aiming for reduced computational cost and memory footprint.

Datasets

The algorithm was trained and evaluated on two standard MVS datasets:

  • DTU Dataset: An indoor multi-view vision dataset with 128 scenes, used for training and testing.
  • Tanks and Temples Dataset: A large-scale outdoor dataset, used to evaluate the algorithm's generalization capabilities on more complex scenes.

Key Insights and Ideas

  • Data Augmentation for Robustness: The paper demonstrates that strategically augmenting training data (e.g., with masks and color transformations) significantly improves the model's robustness to real-world challenges like varying lighting and occlusions, leading to better reconstruction quality.
  • Patchmatch Efficiency: By integrating a learned patchmatch algorithm and replacing heavy 3D convolutions with adaptive propagation and evaluation, the authors achieve substantial reductions in runtime and memory consumption. This is a key enabler for processing higher-resolution images.
  • Dynamic Depth Sampling: The use of dynamic depth sampling intervals, rather than fixed ones, allows the algorithm to capture finer details and more image information, especially across different scales.
  • Coarse-to-Fine Strategy: The multi-stage, coarse-to-fine approach, combined with adaptive propagation, allows for efficient refinement of the depth map, leading to improved accuracy.

Key Results

  • Competitive Accuracy: The proposed algorithm achieves competitive accuracy on both the DTU and Tanks and Temples datasets, often reconstructing finer details and boundaries compared to other state-of-the-art methods. For instance, on the DTU dataset, it achieves an "Overall" score of 0.344 mm, placing it among the top performers.
  • Reduced Runtime and Memory: The algorithm significantly outperforms other learning-based methods in terms of calculation time and memory consumption, as shown in Figure 5. For example, it processes images at 640x512 resolution in 0.18 seconds.
  • Improved Completeness: Qualitative results (Figure 4) show that the algorithm reconstructs more complete point clouds, particularly at the fine structures of boundaries.
  • Generalization: The model trained on the DTU dataset shows good performance on the more complex Tanks and Temples dataset without further fine-tuning, highlighting its generalization ability.

Limitations and Future Work

  • Accuracy vs. Speed Trade-off: While the algorithm excels in speed and memory efficiency, the authors acknowledge a desire to "solve the defect of low accuracy without increasing the calculation consumption." This suggests that further improvements in accuracy are a target for future work.
  • Application to Resource-Constrained Platforms: The paper explicitly mentions the goal of applying the algorithm to "small platforms with limited computing resources, such as wearable devices and mobile phones." Future work will likely focus on optimizing the model further for these environments.
  • Further Accuracy Improvements: The authors express interest in improving accuracy, especially for scenarios where current methods might struggle.

Summary updated

Abstract

In this paper, a multi-view stereo vision patchmatch algorithm based on data augmentation is proposed. Compared to other works, this algorithm can reduce runtime and save computational memory through efficient cascading of modules; therefore, it can process higher-resolution images. Compared with algorithms utilizing 3D cost volume regularization, this algorithm can be applied on resource-constrained platforms. This paper applies the data augmentation module to an end-to-end multi-scale patchmatch algorithm and adopts adaptive evaluation propagation, avoiding the substantial memory resource consumption characterizing traditional region matching algorithms. Extensive experiments on the DTU and Tanks and Temples datasets show that our algorithm is very competitive in completeness, speed and memory.
BibTeX
@inproceedings{pan2023multi,
  author = {Feiyang Pan and Pengtao Wang and Lin Wang and Lihong Li},
  title = {{Multi-View Stereo Vision Patchmatch Algorithm Based on Data Augmentation}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23052729},
}
Figure