PA-MVSNet: Sparse-to-Dense Multi-View Stereo With Pyramid Attention

Ke Zhang, Mengyu Liu, Jinlai Zhang, Zhenbiao DongIEEE Access 2021 · OpenReview

Abstract

Multi-view based 3D reconstruction aims to obtain 3D structure information of objects in space through two-dimensional images. In this paper, we propose a new multi-view stereo network that can robustly reconstruct the scene. To enhance the feature representation ability of Point-MVSNet, a pyramid attention module is introduced. Specifically, we exploit the attention mechanism for the multi-scale feature pyramid to capture larger receptive fields and richer information. Instead of constructing a feature pyramid as the input, results of the pyramid attention module at different scales are directly used for the next layer. The network eventually generates a high-quality depth estimation for 3D reconstruction from sparse to dense by an iterative refinement schedule. Experiments have been performed to evaluate 3D reconstruction quality by comparison with existing state-of-the-art methods on the DTU dataset. The experimental results indicate our method performs the best in overall quality compared with previous methods, proving the effectiveness of our method. In the end, we use the data collected by mobile devices to implement 3D reconstruction with a combination of traditional and learning-based methods, providing ideas for the 3D reconstruction technology on mobile devices.

Introduction

Multi-View Stereo (MVS) is a fundamental task in computer vision, involving the reconstruction of 3D scenes from a collection of 2D images with known camera parameters. While deep learning has significantly advanced MVS, current state-of-the-art methods face two primary challenges. First, many high-performing models rely on 3D cost volumes, which are memory-intensive and limit the achievable resolution of the final 3D model. Second, newer "point-based" methods, while more memory-efficient, often suffer from narrow receptive fields. This "tunnel vision" makes it difficult to capture global context, leading to inaccuracies in textureless regions where local matching is ambiguous.

PA-MVSNet addresses these issues by introducing a Pyramid Attention Module (PAM). This module allows the network to capture long-range dependencies across multiple scales, providing the global context necessary for robust reconstruction without the prohibitive memory costs of 3D CNNs.

Methodology

The architecture of PA-MVSNet follows a sparse-to-dense iterative refinement strategy. The process begins with feature extraction from input images to build a multi-scale feature pyramid. These features are then enhanced through the attention mechanism before being used for depth estimation.

The overall architecture of PA-MVSNet, showing the flow from input images to the refined depth map via the Coarse Depth Network and PointFlow refinement.

Pyramid Attention Module (PAM)

The core innovation is the integration of a Scale-Agnostic Attention mechanism within the feature pyramid. Unlike standard attention that operates within a single resolution, PAM enables the network to find correspondences across different scales. This allows high-resolution details to be informed by low-resolution global structures, resolving local ambiguities.

Detailed view of the Scale-Agnostic Attention mechanism, illustrating how features from different scales (1-scale and 2-scale) are processed and combined.

Sparse-to-Dense Refinement

The network first generates a coarse depth map using a dedicated sub-network. This initial estimate is then refined through an iterative process. By unprojecting 2D features into 3D space to form a point cloud, the model predicts "depth residuals"—the errors in the current estimate—and adjusts the 3D points accordingly. This point-based processing ensures computational efficiency while maintaining high precision.

Experimental Results

The proposed model was evaluated on the DTU Dataset, a standard benchmark for indoor 3D reconstruction consisting of 124 scenes.

  • Quantitative Performance: PA-MVSNet achieved an overall score of 0.375 mm (lower is better), outperforming previous state-of-the-art methods like Point-MVSNet (0.391 mm) and R-MVSNet (0.422 mm). Its accuracy reached 0.313 mm, demonstrating a significant improvement over baseline point-based models.
  • Qualitative Analysis: Visual inspections reveal that PA-MVSNet produces cleaner point clouds with fewer artifacts and less noise compared to traditional methods like COLMAP.

Real-World Application on Mobile Devices

To demonstrate practical utility, the authors implemented a hybrid pipeline for mobile data. Using an iPhone, image sequences were captured and processed. Traditional algorithms (COLMAP) were used for camera pose estimation, while PA-MVSNet handled the dense 3D reconstruction. This approach successfully produced high-quality 3D models from handheld device data, proving the method's robustness outside of controlled laboratory settings.

Conclusion

PA-MVSNet introduces a powerful pyramid attention mechanism to point-based MVS, effectively expanding the receptive field and improving feature representation. By combining global context with an efficient sparse-to-dense refinement strategy, the model achieves state-of-the-art results on benchmark datasets and proves robust for real-world mobile applications. Future work will focus on improving completeness and optimizing the model for real-time inference on mobile hardware.

Summary updated

Abstract

Multi-view based 3D reconstruction aims to obtain 3D structure information of objects in space through two-dimensional images. In this paper, we propose a new multi-view stereo network that can robustly reconstruct the scene. To enhance the feature representation ability of Point-MVSNet, a pyramid attention module is introduced. Specifically, we exploit the attention mechanism for the multi-scale feature pyramid to capture larger receptive fields and richer information. Instead of constructing a feature pyramid as the input, results of the pyramid attention module at different scales are directly used for the next layer. The network eventually generates a high-quality depth estimation for 3D reconstruction from sparse to dense by an iterative refinement schedule. Experiments have been performed to evaluate 3D reconstruction quality by comparison with existing state-of-the-art methods on the DTU dataset. The experimental results indicate our method performs the best in overall quality compared with previous methods, proving the effectiveness of our method. In the end, we use the data collected by mobile devices to implement 3D reconstruction with a combination of traditional and learning-based methods, providing ideas for the 3D reconstruction technology on mobile devices.
BibTeX
@inproceedings{zhang2021pa,
  author = {Ke Zhang and Mengyu Liu and Jinlai Zhang and Zhenbiao Dong},
  title = {{PA-MVSNet: Sparse-to-Dense Multi-View Stereo With Pyramid Attention}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=D7HTha1pki},
}
Figure