GC-MVSNet: Multi-View, Multi-Scale, Geometrically-Consistent Multi-View Stereo

Vibhas K. Vats, Sripad Joshi, David J. Crandall, Md. Alimoor Reza, Soon-heung JungWACV 2024

TL;DR

This paper introduces GC-MVSNet, a novel multi-view stereo (MVS) approach that explicitly enforces geometric consistency during training, not just as a post-processing step. By penalizing geometrically inconsistent pixels across multiple views and scales, GC-MVSNet significantly accelerates learning, requiring nearly half the training iterations of previous state-of-the-art methods, while also achieving new SOTA results on benchmark datasets.

What Problem Does This Paper Address?

Traditional multi-view stereo (MVS) methods aim to reconstruct a 3D scene from multiple 2D images. These methods rely on two core principles: photometric consistency (pixels with the same 3D point should have similar colors) and geometric consistency (the 3D point reconstructed from different views should be the same). While modern deep learning-based MVS methods excel at learning photometric consistency, they often treat geometric consistency as a secondary concern, applying it only after the initial depth estimation. This implicit learning of geometric constraints can be slow and inefficient. The paper addresses the challenge of how to effectively integrate multi-view geometric consistency directly into the learning process to accelerate training and improve accuracy.

Methods

The core innovation of GC-MVSNet is its Geometric Consistency (GC) module, which is applied at each stage of a multi-scale MVS pipeline. This module works as follows:

  • Multi-View Geometric Consistency Check: For each pixel in the reference view's estimated depth map, the module projects it into multiple source views. It then reprojects these points back to the reference view.
  • Inconsistency Penalty: By comparing the original and reprojected depth values, the module identifies pixels that are geometrically inconsistent across views. This inconsistency is quantified by calculating Pixel Displacement Error (PDE) and Relative Depth Difference (RDD).
  • Stage-wise Loss: A binary mask is generated based on these errors, indicating inconsistent pixels. This mask is then used to create a per-pixel penalty ($\xi_p$). This penalty is element-wise multiplied with the per-pixel depth error ($\xi_\alpha$, derived from cross-entropy loss) to form the stage-wise loss ($L_i$).
  • Accelerated Learning: This explicit geometric penalty forces the model to learn geometrically consistent depth maps more quickly, significantly reducing the number of training iterations required.
  • Network Architecture: The paper uses a deformable convolution-based Feature Pyramid Network (FPN) for feature extraction and employs group normalization and weight standardization for stable training.

Datasets

The paper evaluates GC-MVSNet on three standard MVS datasets:

  • DTU: A widely used indoor dataset with 128 scenes, providing ground truth depth maps and point clouds.
  • BlendedMVS: A large-scale synthetic dataset with both indoor and outdoor scenes, primarily used for depth map evaluation.
  • Tanks and Temples: A benchmark dataset featuring more complex and realistic scenes, used for point cloud evaluation.

Key Insights and Ideas

  • Explicit Geometric Consistency During Training is Key: The most significant insight is that directly incorporating geometric consistency checks into the loss function during training, rather than as a post-processing step, dramatically accelerates learning and improves performance.
  • Multi-Scale Approach Enhances Robustness: Applying the GC module at multiple scales (coarse, intermediate, and refinement stages) allows the model to learn geometric consistency at different levels of detail, leading to more robust reconstructions.
  • Generic Module for Plug-and-Play Integration: The GC module is designed to be generic and can be easily integrated into existing MVS pipelines, demonstrating its versatility and broad applicability.
  • Reduced Training Iterations: The explicit geometric supervision leads to a remarkable reduction in training time, often requiring nearly half the iterations of previous methods.

Key Results

  • State-of-the-Art Performance: GC-MVSNet achieves new state-of-the-art results on the DTU and BlendedMVS datasets.
  • Competitive Results: It also demonstrates competitive performance on the Tanks and Temples benchmark.
  • Training Speedup: The paper reports a reduction in training iterations by nearly half compared to other state-of-the-art MVS methods.
  • Qualitative Improvements: Visual comparisons show that GC-MVSNet generates denser and more complete point clouds, particularly in challenging areas.

Limitations and Future Work

  • Hyperparameter Tuning: Like many deep learning models, GC-MVSNet requires careful tuning of hyperparameters, including those for the GC module (e.g., $D_{pixel}$, $D_{depth}$, $M$) and the main MVS pipeline (e.g., depth interval ratio).
  • Ground Truth Quality: The performance of the GC module is dependent on the quality of the ground truth depth maps used for training.
  • Future Work: The authors suggest that their GC module can be plugged into various MVS pipelines, implying future work could explore its integration with a wider range of architectures and techniques. The paper also hints at further research into optimizing the hyperparameter tuning process.

Summary updated

BibTeX
@inproceedings{vats2024gc,
  author = {Vibhas K. Vats and Sripad Joshi and David J. Crandall and Md. Alimoor Reza and Soon-heung Jung},
  title = {{GC-MVSNet: Multi-View, Multi-Scale, Geometrically-Consistent Multi-View Stereo}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure