Multiview Stereo with Cascaded Epipolar RAFT

Zeyu Ma, Zachary Teed, Jia Deng

Abstract

We address multiview stereo (MVS), an important 3D vision task that reconstructs a 3D model such as a dense point cloud from multiple calibrated images. We propose CER-MVS (Cascaded Epipolar RAFT Multiview Stereo), a new approach based on the RAFT (Recurrent All-Pairs Field Transforms) architecture developed for optical flow. CER-MVS introduces five new changes to RAFT: epipolar cost volumes, cost volume cascading, multiview fusion of cost volumes, dynamic supervision, and multiresolution fusion of depth maps. CER-MVS is significantly different from prior work in multiview stereo. Unlike prior work, which operates by updating a 3D cost volume, CER-MVS operates by updating a disparity field. Furthermore, we propose an adaptive thresholding method to balance the completeness and accuracy of the reconstructed point clouds. Experiments show that our approach achieves competitive performance on DTU (the second best among known results) and state-of-the-art performance on the Tanks-and-Temples benchmark (both the intermediate and advanced set).

Introduction

Multiview Stereo (MVS) is the process of reconstructing a dense 3D point cloud from a collection of calibrated 2D images. The dominant paradigm in recent years, established by models like MVSNet, involves building a 3D "cost volume" to represent potential matches across images and then "regularizing" this volume using 3D convolutional neural networks (CNNs).

However, 3D convolutions are computationally expensive and consume massive amounts of GPU memory. This bottleneck often forces a trade-off: researchers must either downsample images or limit depth resolution, which sacrifices fine details in the final 3D model. In this paper, we introduce CER-MVS (Cascaded Epipolar RAFT Multiview Stereo), which shifts the strategy by building on the RAFT architecture. Instead of regularizing a static 3D volume, CER-MVS treats depth estimation as an iterative refinement of a 2D disparity field.

Methodology

CER-MVS adapts the high-performance RAFT architecture—originally designed for optical flow—to the geometric constraints of 3D reconstruction. The core innovation lies in updating a disparity field rather than a 3D cost volume.

Key Architectural Innovations

  • Epipolar Cost Volumes: Unlike the 4D volumes used in motion estimation, CER-MVS uses camera geometry to restrict the search to the epipolar line. It constructs a 3D cost volume where one dimension represents increments of inverse depth (disparity).
  • Iterative Disparity Updates: A Gated Recurrent Unit (GRU) iteratively updates a 2D disparity field. In each step, the GRU "looks up" values from the cost volume based on its current estimate to decide how to refine the depth.
  • Cost Volume Cascading: To handle high-resolution details without exploding memory usage, the model uses a coarse-to-fine approach. After initial iterations, it builds a new, finer-grained cost volume centered specifically around the current depth predictions.
  • Multiview and Multiresolution Fusion: The system aggregates information from multiple neighbor views by taking the element-wise mean of their individual cost volumes. Additionally, a novel heuristic fuses depth maps generated at different resolutions, using low-resolution maps for structural stability and high-resolution maps for fine details.

Adaptive Thresholding

Standard point cloud "stitching" (combining multiple depth maps) often relies on hard-coded thresholds that do not generalize well across different scenes. We propose an adaptive thresholding method that ensures a fixed percentage of pixels pass consistency checks, dynamically balancing accuracy and completeness.

Experiments and Results

The model was trained on the BlendedMVS synthetic dataset and evaluated on two primary benchmarks: DTU and Tanks-and-Temples.

State-of-the-Art Performance

CER-MVS achieved the #1 spot on the Tanks-and-Temples benchmark for both the Intermediate (Mean F1: 64.82) and Advanced (Mean F1: 40.19) sets at the time of publication. This benchmark is considered the "gold standard" for testing how well MVS algorithms generalize to complex, real-world environments.

Precision and Efficiency

On the DTU dataset, CER-MVS achieved the second-best overall score among published methods (0.332 mm), demonstrating high precision. Furthermore, the cascading mechanism allows the model to achieve high-resolution results with significantly less memory than non-cascaded equivalents, proving that updating a 2D field is computationally superior to 3D volume regularization.

Discussion and Limitations

While CER-MVS sets a new standard for MVS performance, some challenges remain. Like most MVS methods, it struggles with low-texture surfaces (e.g., blank walls) where the cost volume provides little signal. Additionally, the multiresolution fusion currently relies on manually designed heuristics; future work could explore learning this fusion step end-to-end. Finally, integrating the point cloud stitching process more tightly into the neural network remains an open challenge.

Conclusion

CER-MVS demonstrates that the iterative refinement principles of RAFT can be successfully ported to 3D vision. By leveraging epipolar geometry and a cascaded architecture, we provide a memory-efficient and highly accurate solution for multiview stereo that excels in both controlled and real-world environments.

Summary updated

Abstract

We address multiview stereo (MVS), an important 3D vision task that reconstructs a 3D model such as a dense point cloud from multiple calibrated images. We propose CER-MVS (Cascaded Epipolar RAFT Multiview Stereo), a new approach based on the RAFT (Recurrent All-Pairs Field Transforms) architecture developed for optical flow. CER-MVS introduces five new changes to RAFT: epipolar cost volumes, cost volume cascading, multiview fusion of cost volumes, dynamic supervision, and multiresolution fusion of depth maps. CER-MVS is significantly different from prior work in multiview stereo. Unlike prior work, which operates by updating a 3D cost volume, CER-MVS operates by updating a disparity field. Furthermore, we propose an adaptive thresholding method to balance the completeness and accuracy of the reconstructed point clouds. Experiments show that our approach achieves competitive performance on DTU (the second best among known results) and state-of-the-art performance on the Tanks-and-Temples benchmark (both the intermediate and advanced set). Code is available at https://github.com/princeton-vl/CER-MVS
BibTeX
@inproceedings{ma2022multiview,
  author = {Zeyu Ma and Zachary Teed and Jia Deng},
  title = {{Multiview Stereo with Cascaded Epipolar RAFT}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19821-2\_42},
}
Figure