MVS2: Deep Unsupervised Multi-View Stereo with Multi-View Symmetry

Yuchao Dai, Zhidong Zhu, Zhibo Rao, Bo Li

Abstract

MVS$^2$ is a novel deep learning framework for high-quality multi-view stereo (MVS) reconstruction that operates without ground-truth 3D training data. By employing a symmetric architecture that predicts depth for all camera views simultaneously and a cross-view consistency loss, the model learns 3D geometry directly from raw images. This approach addresses the "data hunger" of supervised models and the inherent asymmetries in traditional deep MVS networks.

Introduction

Multi-view Stereo (MVS) aims to reconstruct 3D scenes from 2D image collections. While deep learning models like MVSNet have surpassed traditional geometric methods in accuracy, they typically require massive amounts of ground-truth depth data (e.g., from LiDAR), which is difficult to acquire and limits generalization. Furthermore, most existing networks are asymmetric, prioritizing a single reference view and ignoring the natural symmetry of 3D space. MVS$^2$ addresses these issues by introducing an unsupervised, symmetric framework that treats all viewpoints equally.

Methodology

The MVS$^2$ architecture is built on three core pillars designed to enable unsupervised learning:

  1. Multi-View Symmetry: Unlike asymmetric models, MVS$^2$ treats all input images equally. It extracts features and constructs cost volumes for every view simultaneously, ensuring that predicted depth maps are inherently compatible.
  2. Cross-View Consistency Loss: The primary supervisory signal is a geometric loss that penalizes inconsistencies between views. By projecting points from one camera's depth map to another and back, the network is forced to converge on a consistent 3D structure.
  3. Integrated Occlusion Reasoning: To handle occlusions, MVS$^2$ utilizes its consistency checks to automatically detect and mask regions where points are visible in one view but blocked in another, preventing erroneous gradients during training.
The MVS$^2$ symmetric architecture processes multiple images simultaneously to ensure geometric consistency across views.

The pipeline utilizes 3D CNNs for cost volume regularization and a Spatial Propagation Network (SPN) for depth map refinement.

Experiments and Results

The model was evaluated on the DTU dataset and several generalization sets, including SUN3D, RGBD, and the Tanks and Temples benchmark.

Key Results

  • Accuracy: On the DTU benchmark, MVS$^2$ achieved a completeness score of 0.515mm, outperforming several supervised methods and traditional algorithms like Camp.
  • Generalization: Without fine-tuning, MVS$^2$ outperformed the COLMAP algorithm on RGBD and SUN3D datasets, demonstrating superior robustness compared to supervised models like DeMoN.
  • Ablation Study: The consistency loss proved vital; its removal led to a 2.5x increase in Absolute Relative Error (from 0.0147 to 0.0355).
Evolution of the learned masks over training steps, showing how the model progressively identifies consistent geometric regions.

Discussion and Conclusion

MVS$^2$ demonstrates that photometric error alone is insufficient for deep MVS; geometric depth consistency is the critical ingredient for unsupervised convergence. By enforcing symmetry, the model reduces the need for complex post-processing. Future work may extend these consistency checks to image "cliques" and address challenges in dynamic scenes and memory constraints associated with symmetric processing.

Summary updated

BibTeX
@inproceedings{dai2019mvs,
  author = {Yuchao Dai and Zhidong Zhu and Zhibo Rao and Bo Li},
  title = {{MVS2: Deep Unsupervised Multi-View Stereo with Multi-View Symmetry}},
  booktitle = {3DV},
  year = {2019},
  doi = {10.1109/3dv.2019.00010},
}
Figure