Multi-View Stereo Using Perspective-Aware Features and Metadata to Improve Cost Volume

Zongcheng Zuo, Yuanxiang Li, Yu Zhou, Fan MoSensors 2025 · DOI

TL;DR

This paper introduces PAC-MVSNet, a novel multi-view stereo (MVS) approach that significantly improves 3D reconstruction by integrating "perspective-aware convolution" (PAC) and metadata into the cost volume. This allows the network to better handle challenging scenes with reflective or texture-less surfaces and achieve state-of-the-art results with a more efficient 2D CNN architecture.

What Problem Does This Paper Address?

Reconstructing accurate 3D models from multiple images (Multi-View Stereo or MVS) is a fundamental problem in computer vision. While deep learning methods have made great strides, they often struggle with challenging scenarios like reflective surfaces, texture-less regions, and occlusions. A key bottleneck is feature matching – finding corresponding points across different images – and the subsequent construction of a "cost volume" that represents the likelihood of a point being at a certain depth. Existing methods often rely on standard convolutional layers that capture local features but miss broader contextual information, and they typically neglect valuable geometric metadata (like camera poses and distances) that humans naturally use to infer depth. This paper aims to bridge this gap by developing a more robust feature extraction and cost volume construction mechanism.

Methods

The core of PAC-MVSNet lies in two novel components:

  • Perspective-Aware Convolution (PAC): Standard convolutional kernels are fixed. PAC, however, dynamically adjusts its kernel shape to align with "perspective lines" derived from the camera's intrinsic parameters. This allows it to better capture features along the depth axis of objects, providing richer cues for depth estimation, especially in scenes with varying viewpoints. It achieves this by using a multi-branch architecture with different dilation rates and an angled kernel form.
  • Feature Matching with Long-Range Tracking (FMLT) and Metadata Integration: The FMLT module uses Transformer's attention mechanisms (both intra-image and inter-image) to capture long-range contextual information, addressing the limitations of local feature extractors. Crucially, the authors integrate readily available metadata (such as camera pose distances and validity masks) directly into the cost volume. This metadata provides geometric reasoning capabilities, allowing the network to better understand spatial relationships and make more informed depth predictions, especially in challenging scenarios.

Instead of relying on computationally expensive 3D convolutions for cost volume regularization, PAC-MVSNet uses a more efficient 2D CNN architecture, making it suitable for resource-constrained environments.

Datasets

The authors evaluated their method on several standard MVS benchmarks:

  • DTU Dataset: A well-controlled dataset with 128 scanned point clouds and 49 camera viewpoints, commonly used for training and testing MVS networks.
  • Tanks and Temples Dataset: A challenging benchmark featuring real-world scenes with diverse and complex geometries and textures, suitable for evaluating robustness.
  • ETH3D Dataset: A high-definition dataset with 12 test scenes, including intricate indoor and outdoor environments, used to assess resilience.
  • UrbanScene3D Dataset: A large-scale dataset focused on urban environments, providing a comprehensive testbed for 3D reconstruction of complex scenes.

Key Insights and Ideas

  • Metadata is Crucial for Geometric Reasoning: Humans infer depth using more than just visual features; they consider relative distances and object relationships. By embedding metadata like camera pose distances into the cost volume, the network gains this geometric reasoning ability, leading to more accurate depth predictions.
  • Perspective Matters: Standard convolutions are viewpoint-agnostic. PAC's ability to align feature extraction with the scene's perspective lines is key to understanding spatial relationships and improving accuracy, especially when dealing with multiple views of the same object.
  • 2D CNNs Can Be Powerful: The paper demonstrates that by intelligently integrating features and metadata, efficient 2D CNNs can achieve performance comparable to more complex 3D convolutional approaches for MVS, making the method more practical.
  • Transformer for Long-Range Context: Leveraging Transformer's attention mechanisms in FMLT effectively captures global context within and across images, which is vital for robust feature matching in complex scenes.

Key Results

  • State-of-the-Art Performance: PAC-MVSNet consistently outperformed existing methods across all evaluated datasets (DTU, Tanks and Temples, ETH3D, UrbanScene3D) on various metrics like mean distance, percentage of points within error thresholds, and F-score.
  • Improved Accuracy and Completeness: The method achieved a strong balance between accuracy and completeness, particularly excelling in challenging areas like texture-less regions and complex geometries. For example, on the DTU dataset, their combined PAC + Metadata method achieved the best F-score of 83.47.
  • Efficiency: The use of 2D CNNs and optimized attention mechanisms resulted in competitive inference speeds and reduced memory consumption compared to other advanced methods, especially at high resolutions. For instance, at 1536x1152 resolution, PAC-MVSNet reduced memory consumption by up to 37.78% and run time by up to 223.43% compared to some baselines.

Limitations and Future Work

  • Occlusion and Extreme Elevation: The authors acknowledge that PAC-MVSNet still faces challenges in highly occluded regions and scenes with extreme elevation variations (like aerial views), partly due to the initial ground plane assumption.
  • Future Work: They propose to address these limitations by integrating elevation priors and developing occlusion-aware attention mechanisms. Additionally, they plan to explore integrating multi-modal information, such as optical flow, to further enhance cross-modal perspective alignment for improved 3D reconstruction.

Summary updated

Abstract

Feature matching is pivotal when using multi-view stereo (MVS) to reconstruct dense 3D models from calibrated images. This paper proposes PAC-MVSNet, which integrates perspective-aware convolution (PAC) and metadata-enhanced cost volumes to address the challenges in reflective and texture-less regions. PAC dynamically aligns convolutional kernels with scene perspective lines, while the use of metadata (e.g., camera pose distance) enables geometric reasoning during cost aggregation. In PAC-MVSNet, we introduce feature matching with long-range tracking that utilizes both internal and external focuses to integrate extensive contextual data within individual images as well as across multiple images. To enhance the performance of the feature matching with long-range tracking, we also propose a perspective-aware convolution module that directs the convolutional kernel to capture features along the perspective lines. This enables the module to extract perspective-aware features from images, improving the feature matching. Finally, we crafted a specific 2D CNN that fuses image priors, thereby integrating keyframes and geometric metadata within the cost volume to evaluate depth planes. Our method represents the first attempt to embed the existing physical model knowledge into a network for completing MVS tasks, which achieved optimal performance using multiple benchmark datasets.
BibTeX
@inproceedings{zuo2025multi,
  author = {Zongcheng Zuo and Yuanxiang Li and Yu Zhou and Fan Mo},
  title = {{Multi-View Stereo Using Perspective-Aware Features and Metadata to Improve Cost Volume}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25072233},
}
Figure