RayMVSNet++: Learning Ray-Based 1D Implicit Fields for Accurate Multi-View Stereo

Yifei Shi, Junhua Xi, Dewen Hu, Zhiping Cai, Kai Xu

Introduction

Multi-View Stereo (MVS) is a fundamental task in computer vision, aimed at reconstructing dense 3D geometry from a set of calibrated 2D images. Traditionally, learning-based MVS methods have relied on 3D Convolutional Neural Networks (CNNs) operating on dense 3D cost volumes. While these methods have achieved significant success, they are inherently limited by the cubic memory and computational complexity of 3D grids. This often necessitates a trade-off between reconstruction resolution and memory feasibility, or the use of complex multi-stage coarse-to-fine architectures.

In "RayMVSNet++: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo," Shi et al. propose a paradigm shift. Instead of regularizing a heavy 3D volume, the authors suggest optimizing depth directly along 1D camera rays. By treating the depth estimation as a sequence modeling problem along rays, RayMVSNet++ achieves high-resolution reconstruction with significantly reduced memory overhead and improved robustness in challenging environments.

The Problem: The 3D Cost Volume Bottleneck

The standard approach in deep MVS involves warping image features into a 3D reference frustum to build a cost volume. This volume is then processed using 3D convolutions to regress depth. However, this approach suffers from two major drawbacks:

  1. Computational Inefficiency: 3D convolutions scale poorly with resolution, often forcing models to operate on downsampled volumes.
  2. Wasted Computation: A significant portion of a 3D cost volume represents empty space or occluded regions. Since the final output is a view-dependent depth map, much of the volumetric processing is redundant.

RayMVSNet++ addresses these issues by focusing computational resources specifically on the rays emanating from the camera, effectively transforming a 3D problem into a series of 1D optimization tasks.

Methodology

RayMVSNet++ employs a two-stage pipeline designed for both efficiency and precision.

1. Coarse Initialization

The process begins with a lightweight, low-resolution 3D cost volume. This stage provides a rough initial depth estimate, which serves as a starting point for the subsequent ray-based refinement. This ensures the model has a global context of the scene geometry before focusing on fine-grained details.

2. Ray-based Refinement and 1D Implicit Fields

For every pixel in the reference image, the model casts a ray and samples points around the coarse depth estimate. The refinement process consists of three core components:

  • Epipolar Transformer: To handle cross-view matching, the model uses an attention mechanism. Instead of simple feature averaging, the Epipolar Transformer identifies the most relevant features along epipolar lines in source views. This allows the network to dynamically weight features, effectively ignoring occlusions or specular reflections that would otherwise corrupt the matching cost.
  • 1D Implicit Field (LSTM): The authors model the Signed Distance Function (SDF) along each ray as a 1D implicit field. An Long Short-Term Memory (LSTM) network processes the sampled points sequentially. This approach leverages the physical prior that SDF values should be monotonic as a ray approaches and passes through a surface, providing a more stable optimization target than direct depth regression.
  • Local-Frustum Context Aggregation: A common pitfall of purely ray-based methods is the lack of horizontal (spatial) context, which is vital for reconstructing textureless regions. RayMVSNet++ introduces an attentional gating unit using the Gumbel-Softmax trick. This allows the model to selectively aggregate features from neighboring rays within a local frustum. By learning which neighbors are semantically relevant, the model can maintain sharp edges while smoothing out noise in low-texture or poorly lit areas.

Experimental Results

The model was rigorously evaluated on several benchmarks, demonstrating state-of-the-art performance across diverse scenarios.

Quantitative Performance

  • DTU Dataset: RayMVSNet++ achieved an overall reconstruction score of 0.328mm, surpassing established models like CasMVSNet (0.355mm) and PatchmatchNet (0.352mm). The model showed particular strength in the authors' custom subsets for specular reflections and occlusions.
  • ScanNet: In indoor environments characterized by motion blur and textureless walls, the model reached a state-of-the-art AbsRel of 0.058m.
  • Tanks & Temples: The model demonstrated strong generalization capabilities on large-scale outdoor scenes, proving that the ray-based approach is robust beyond controlled laboratory settings.

Qualitative Advantages

Qualitatively, RayMVSNet++ produces reconstructions with significantly sharper boundaries and better completeness on thin structures, such as chair legs or complex machinery. The local-frustum context aggregation proves particularly effective at filling in gaps in textureless regions where traditional volume-based methods often produce artifacts or holes.

Conclusion and Future Work

RayMVSNet++ represents a significant step forward in making Multi-View Stereo more efficient and accurate. By rethinking the cost volume as a collection of 1D implicit fields, the method bypasses the memory bottlenecks of 3D CNNs while maintaining high reconstruction quality.

Future research directions include eliminating the dependency on the initial coarse 3D volume to create a fully ray-based pipeline, and integrating learnable camera pose estimation to handle real-world data where perfect calibration is unavailable. Additionally, the authors suggest that extending this ray-based logic to direct 3D point cloud optimization could further improve the end-to-end reconstruction workflow.

Summary updated

Abstract

Learning-based multi-view stereo (MVS) has by far centered around 3D convolution on cost volumes. Due to the high computation and memory consumption of 3D CNN, the resolution of output depth is often considerably limited. Different from most existing works dedicated to adaptive refinement of cost volumes, we opt to directly optimize the depth value along each camera ray, mimicking the range (depth) finding of a laser scanner. This reduces the MVS problem to ray-based depth optimization which is much more light-weight than full cost volume optimization. In particular, we propose RayMVSNet which learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. This sequential modeling, conducted based on transformer features, essentially learns the epipolar line search in traditional multi-view stereo. We devise a multi-task learning for better optimization convergence and depth accuracy. We found the monotonicity property of the SDFs along each ray greatly benefits the depth estimation. Our method ranks top on both the DTU and the Tanks & Temples datasets over all previous learning-based methods, achieving an overall reconstruction score of 0.33 mm on DTU and an F-score of 59.48% on Tanks & Temples. It is able to produce high-quality depth estimation and point cloud reconstruction in challenging scenarios such as objects/scenes with non-textured surface, severe occlusion, and highly varying depth range. Further, we propose RayMVSNet++ to enhance contextual feature aggregation for each ray through designing an attentional gating unit to select semantically relevant neighboring rays within the local frustum around that ray. This improves the performance on datasets with more challenging examples (e.g., low-quality images caused by poor lighting conditions or motion blur). RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset. In particular, it attains an AbsRel of 0.058m and produces accurate results on the two subsets of textureless regions and large depth variation.
BibTeX
@article{shi2023raymvsnet,
  author = {Yifei Shi and Junhua Xi and Dewen Hu and Zhiping Cai and Kai Xu},
  title = {{RayMVSNet++: Learning Ray-Based 1D Implicit Fields for Accurate Multi-View Stereo}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=vuaqHfc_Bk_},
}
Figure