Multi-View Stereo Representation Revist: Region-Aware MVSNet

Yisu Zhang, Jianke Zhu, Lixiang LinCVPR 2023 · DOI

Introduction

Multi-View Stereo (MVS) is a fundamental task in computer vision, aimed at reconstructing 3D geometry from multiple 2D images. Traditional deep-learning-based MVS methods typically formulate this as a pixel-wise depth estimation problem. For each camera ray, the model predicts a single depth value representing the surface intersection. While effective, this point-based representation often fails in textureless regions where visual matching is ambiguous and near object boundaries where foreground-background confusion leads to "flying" noise points.

RA-MVSNet (Region-Aware MVSNet) improves 3D reconstruction by moving beyond simple point-on-ray depth estimation. By incorporating a Signed Distance Function (SDF) branch alongside traditional depth prediction, the model becomes "region-aware," significantly improving accuracy in tricky textureless areas and reducing noise at object boundaries.

The Region-Aware Approach

The core limitation of standard MVS is the neglect of surface topology. By treating surfaces as collections of independent points, models lose the structural context necessary to resolve ambiguities. RA-MVSNet addresses this by shifting from a point-based search to a region-aware representation.

Figure 6: The region-aware representation associates a hypothetical plane point with a local surface patch, allowing the model to perceive geometry beyond a single ray intersection.

Methodology

RA-MVSNet introduces a dual-branch architecture built upon the Cascade MVSNet framework. Instead of a single probability volume, it constructs two distinct volumes:

  1. Probability Volume: Estimates the likelihood of a surface intersection at specific depth intervals.
  2. Distance Volume: Estimates the Signed Distance Function (SDF), representing the distance from a sampled point to the nearest surface point.

By supervising the model with the distance to the nearest surface point—rather than just the intersection point on the current ray—the network learns to perceive local geometry. This distance volume acts as a natural denoiser; points identified as far from the surface are suppressed in the final depth calculation, effectively pruning outliers at object boundaries. To maintain efficiency, the authors developed a patch-based nearest neighbor search to calculate ground-truth distances during training.

Experimental Results

The model was evaluated on the DTU, BlendedMVS, and Tanks and Temples datasets. RA-MVSNet achieved state-of-the-art performance, ranking first in completeness (0.268mm) and overall score (0.297mm) on the DTU benchmark, outperforming models like CasMVSNet and Uni-MVSNet.

Figure 3: Qualitative results of RA-MVSNet showing a clean and complete 3D reconstruction of a complex horse statue.

On the Tanks and Temples benchmark, the model achieved the highest F-scores for both Intermediate (65.72) and Advanced (39.93) sets. Qualitatively, the approach produces significantly cleaner boundaries and more complete surfaces in featureless areas compared to previous point-based methods, as seen in the reconstruction of complex objects like the horse statue.

Conclusion

RA-MVSNet demonstrates that incorporating structural information through an SDF branch significantly enhances MVS performance. By moving from points to patches, the model achieves superior robustness in challenging scenarios. While the dual-branch approach adds computational overhead, the resulting improvements in topology and denoising are substantial. Future work may explore simplifying the pipeline by relying solely on the SDF branch for reconstruction.

Summary updated

BibTeX
@inproceedings{zhang2023multi,
  author = {Yisu Zhang and Jianke Zhu and Lixiang Lin},
  title = {{Multi-View Stereo Representation Revist: Region-Aware MVSNet}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01667},
}
Figure